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Abstract 


Science  and  technology  policy  continues  to  be  limited  by  how  little  is  known  about  the 
drivers  of  innovation.  One  barrier  in  conducting  systematic  studies  is  the  lack  of  an  objective 
measure  for  innovation.  Patents  continue  to  be  an  attractive  measurement  tool,  but  many 
questions  remain  about  their  comprehensiveness,  relevance  in  different  domains,  and  accuracy 
given  the  highly  skew  med  distributions  seen  in  different  estimates  of  patent  value.  This  study 
develops  a  new  approach  to  measuring  research  and  innovation  performance  using  patents  by 
examining  the  trends  in  patent  filings  over  time  within  organizations  and  within  technology 
classes.  Within  any  single  organization’s  patent  portfolio,  the  sequence  of  patent  filings  over 
time  in  any  given  class  tends  to  follow  one  of  four  patterns.  These  within-organization,  within- 
class  patterns  are  potentially  signatures  of  specific  research  and  commercialization  approaches 
which  have  innovative  connotations.  This  study  develops  several  hypotheses  regarding  the 
organizational  drivers  of  these  patenting  patterns  and,  using  data  from  the  DOD  laboratories, 
demonstrates  how  these  patenting  patterns  can  be  used  to  study  the  relationships  between  the  rate 
and  type  of  innovation  and  various  quantitative  and  qualitative  organizational  characteristics. 
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Chapter  1  Introduction 


“  ...we  [need]  to  reach  a  level  of  research  and  development  we  haven ’t  seen  since  the 
height  of  the  Space  Race...  We  11  invest  in  biomedical  research,  information  technology, 
and  especially  clean  energy  technology  —  an  investment  that  will  strengthen  our  security, 
protect  our  planet,  and  create  countless  new  jobs  for  our  people.  ”  President  Barack 
Obama  (2011),  State  of  the  Union  Address 


1.1  Background 

Improving  the  performance  of  the  national  innovation  system  has  been  and  continues  to  be  an 
active  area  of  public  policy.1  While  one  of  the  prototypical  examples  of  a  national  innovation 
program  is  the  1960s  Apollo  Program,2  the  Space  Race  was  not  necessarily  the  height  of  U.S. 
research  and  development  (R&D)  investment.  Federal  support  for  R&D-driven  innovation  has 
remained  high  throughout  the  last  half  of  the  20th  century  and  the  beginning  of  the  21st.  Since  the 
Apollo  Program’s  peak,  total  federal  R&D  funding  has  grown  by  50%,  from  about  $87  billion  in 
fiscal  year  1966  to  $130  billion  in  fiscal  year  201 1  (in  constant  2012  dollars).3  This  funding  has 
supported  several  high-profile  Big  Science  programs,4  such  as  the  War  on  Cancer  and  the 
Human  Genome  Project  that  focus  resources  on  particular  scientific  and  technological 
challenges,  as  well  as  national  laboratories  and  individual  research  grants  that  aim  to  support 
national  defense,  protect  the  environment,  and  accelerate  the  economy  in  general.5 & 


1  A  national  innovation  system  is  the  set  of  actors  within  a  nation  who  produce  or  influence  the  production  of  new 
technologies,  including  private  companies,  universities,  and  government  institutions,  and  the  relationships  among 
them  (see  summary  in  OECD  (1997,  pp.  9-10)). 

2 

A  second  prototypical  example  is  the  Manhattan  Project. 

3 

Based  on  analysis  of  American  Association  for  the  Advancement  of  Science  (AAAS)  data  table  “Federal  R&D 
Funding  by  Performer,  FY  1955-2011”,  available  at  http://www.aaas.org/spp/rd/guihist.shtml  (accessed  2/9/2013). 
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Big  Science  refers  to  “projects  that  [require]  large-scale  organization,  massive  commitments  of  funds,  and 
complex  technological  systems,”  (Capshew  &  Rader,  1992,  p.  4).  The  origin  of  the  term  is  generally  attributed  to 
Weinberg  (1961)  and  de  Solla  Price  (1963),  although  the  phrase  appears  at  least  once  in  the  1950s  as  well  (Capshew 

&  Rader,  1992;  Mack,  1998).  The  Big  Science  concept  has  been  used  to  describe  unitary  crash  technology  programs 
like  the  Apollo  Program  or  the  Human  Genome  Project  (e.g.  Fambright  (2002)),  investments  in  large  and  expensive 
scientific  instruments  like  the  Superconducting  Super  Collider  or  the  Targe  Hadron  Collider  (e.g.  Office  of 
Technology  Assessment  (1991)),  as  well  as  the  growth  in  the  scale  of  the  global  scientific  enterprise  since  the  1600s 
(de  Solla  Price,  1963). 

The  term  national  laboratory,  as  used  in  this  sentence,  covers  all  in-house  government  research  facilities  including 
those  run  by  the  Department  of  Energy  (DOE),  the  Department  of  Defense  (DOD),  the  Department  of  Agriculture, 
the  National  Aeronautics  and  Space  Administration  (NASA),  and  the  Department  of  Health  and  Human  Services 
(HHS).  The  term  individual  research  grants  refers  to  funding  sent  to  university  and  other  private-sector  researchers 
by  government  agencies  including  the  National  Science  Foundation  (NSF),  the  National  Institutes  of  Health  (NIH), 
and  those  listed  in  the  previous  sentence. 
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Today,  there  is  a  particular  focus  on  using  federal  R&D  policy  to  speed  the  development  of 
renewable  energy  technologies  and  advance  the  competitiveness  of  American  industries.  Ten 
different  bills  calling  for  a  Manhattan  Project  or  Apollo  Program  for  green  energy  have  been 
introduced  in  Congress  since  2005, 6  and  the  need  for  more  research-driven  innovation  to  support 
the  environment  or  the  economy  has  been  a  theme  in  seven  of  the  last  eight  Presidential  State  of 

n 

the  Union  addresses. 

However,  these  policy  problems  -  the  need  for  renewable  energy  and  competitive  industries 

O 

-  require  Big  Innovation,  not  necessarily  Big  Science.  If  Big  Science  is  the  large-scale 
applications  of  resources  to  obtain  new  knowledge,9  then  Big  Innovation  is  the  large-scale 
application  of  resources  to  obtain  inventions  which  are  used  to  solve  large-scale  problems. 
Policies  that  enable  one  do  not  necessarily  advance  the  other. 

Most  science  and  innovation  policy-making  assumes  that  if  more  resources  are  provided  to 
science,  particularly  university-  and  laboratory-based  basic  science,  more  economically  and 
socially  significant  innovations  inevitably  result  (Averch,  1985;  Jensen,  Johnson,  Lorenz,  & 
Lundvall,  2007;  Stokes,  1997). 10  Yet  a  number  of  unanswered  questions  remain  about  how  this 
chain  from  laboratory  research  to  implemented  innovation  actually  works.  For  example,  how 
much  basic  research  is  sufficient  to  supply  the  rest  of  the  research  and  development  pipeline  with 
fundamental  knowledge?11  How  can  the  rate  at  which  knowledge  is  implemented  in  inventions 
be  increased?  What  is  the  relationship  between  the  goals  of  a  national  innovation  program  and 
the  most  effective  organizational  format  for  the  program?  Furthermore,  must  all  research 
organizations  endeavor  to  be  radical  innovators  or  should  some  focus  on  incremental  advances  of 
the  state-of-the-art? 


6  Relevant  bills  identified  by  reading  summaries  of  bills  returned  by  a  search  for  the  terms  “Manhattan  Project”  or 
“Apollo  Program”  in  the  Congressional  Record  at  the  Library  of  Congress.  See  for  example  (U.S.  House  of 
Representatives,  2007a,  2007b,  2008a,  2008b,  2008c,  2008d,  2009,  2011;  U.S.  Senate,  2005,  2009) 

7 

Based  on  keyword-in-context  analyses  of  the  terms  “innovation”  or  “innovative”  in  the  C-SPAN  State  of  the 
Union  transcript  archive 

8 

Mack  (1998)  notes  that  “the  confusion  between  science  and  technology  starts  not  in  the  minds  of  historians  writing 
about  the  projects  but  in  the  minds  of  the  policymakers  and  scientists  who  shaped  and  advised  these  projects.” 

9 

See  footnote  4. 

10  The  cited  books  provide  evidence  of  this  trend  in  policy-making;  they  do  not  necessarily  agree  that  it  is  the  best 
approach. 

11  As  former  director  of  the  White  House  Office  of  Science  and  Technology  Policy,  John  Marburger,  noted, 
“Perhaps  the  U.S.  is  spending  as  much  on  federal  R&D  as  it  needs  to,  perhaps  more,  perhaps  less.  Undoubtedly  it 
could  be  spending  it  more  wisely,”  (Marburger,  2007,  p.  29). 

12 

Nor  is  there  even  consensus  on  how  well  past  national  innovation  programs  have  performed.  For  example,  some 
argue  that  the  War  on  Cancer  has  delivered  over  a  trillion  dollars  in  value  to  cancer  patients  through  higher  survival 
rates  (Sun  et  al.,  2009),  while  others  point  out  that  those  gains  have  been  restricted  to  only  a  few  types  of  cancer, 
resulting  in  no  overall  gains  for  the  population  (Faguet,  2005).  A  recent  line  of  criticism  argues  that  an  overly 
conservative  proposal  review  process  at  NIH  has  blocked  revolutionary  research  proposals  that  could  have  delivered 
larger  breakthroughs  (Begley,  2008;  Faguet,  2005;  Leaf,  2007;  Ness,  2010). 
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1  ^ 

Despite  a  vast  scientific  literature  that  discusses  innovation,  the  understanding  of 
innovation  remains,  as  John  Kao  put  it,  in  a  pre-Copemican  state: 

"It's  as  if  we  don't  yet  know  which  heavenly  bodies  revolve  around  which  others.  We  don't 
even  know  where  all  the  planets  are  located,  nor  do  we  have  a  viable  theory  of  planetary 
motion.  We  rely  on  metaphors  and  images  to  express  an  as  yet  imprecise  and 
unsystematic  understanding. "  (Kao,  2007,  pp.  16-17) 

Alternately,  the  current  understanding  of  research  and  innovation  could  be  called  pre-Newtonian. 
Thomas  Kuhn  wrote  of  pre-Newtonian  physics, 

‘‘Being  able  to  take  no  common  body  of  belief for  granted,  each  writer ...  felt  forced  to 
build  his  field  anew  from  its  foundations.  In  doing  so,  his  choice  of  supporting 
observation  and  experiment  was  relatively  free,  for  there  was  no  standard  set  of  methods 
or  of phenomena  that  every...  writer  felt  forced  to  employ  and  explain.  ”  (Kuhn,  1962,  p. 

13) 

Applying  either  metaphor,  the  challenge  for  those  who  would  make  innovation  policy  a  more 
deliberate  tool  is  to  identify  the  key  variables,  and  the  relationships  among  them,  that  govern  the 
innovation  process.  A  logical  place  to  start  is  to  define  innovation  and  determine  a  way  to 
measure  it.  With  no  common  definition  of  innovations,  researchers  are  currently  relatively  free  to 
choose  their  own  supporting  observation  and  experiment.14  A  standard  measurement  for 
innovation  would  enable  researchers  to  compare  different  organizations  along  this  measure  and 
isolate  factors  that  seem  to  be  correlated  with  higher  or  lower  levels  of  innovation.  With  repeated 
testing,  these  approaches  could  become  the  standard  set  of  methods  that  lead  to  a  systematic 
understanding  of  the  innovation  process. 

1 .2  Approach 

" In  this  desert  of  data ,  patent  statistics  loom  up  as  a  mirage  of  wonderful  plentitude  and 
objectivity.  They  are  available;  they  are  by  definition  related  to  inventiveness,  and  they 
are  based  on  what  appears  to  be  an  objective  and  only  slowly  changing  standard.  No 
wonder  that  the  idea  that  something  interesting  might  be  learned  from  such  data  tends  to 
be  rediscovered  in  each  generation. "  (Griliches,  1990) 


13 

For  a  sense  of  scale,  consider  that  a  google  scholar  search  for  the  keyword  “innovation”  on  10/29/2012  returned 
“About  2,360,000”  hits. 

14 

For  example,  NSF’s  2008  Science  of  Science  Policy  Roadmap  recommends  measuring  innovation  as  an  outcome 
of  scientific  research,  but  it  neither  defines  innovation  nor  states  how  such  a  measurement  could  be  done  (Valdez  & 
Lane,  2008).  The  Oslo  Manual,  the  Organization  for  Economic  Co-operation  and  Development’s  (OECD)  guide  to 
surveying  firms  about  innovation,  defines  a  technological  product  innovation  as  a  product  that  is  new  (to  the  firm) 
and  has  been  offered  for  sale.  The  innovation  may  be  new  because  it  uses  a  completely  new  technology  or  because  it 
combines  existing  technologies  in  a  new  way  to  achieve  new  functionality.  However,  innovation  measurement 
techniques  descried  in  the  Oslo  Manual  are,  perforce,  focused  on  surveying  firms  directly  about  their  innovative 
activities. 
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To  move  forward  the  development  of  a  standard  measure  of  innovation,  this  dissertation 
develops  several  new  patent-based  tools  for  monitoring  the  performance  of  research 
organizations  and  identifying  potential  innovations  among  their  outputs.  The  patent-based 
measures  discussed  in  this  dissertation  can  be  used  to  identify  specific  innovations  produced  by 
research  groups  within  a  laboratory,  to  monitor  the  fit  between  the  laboratory’s  outputs  (both 
innovative  and  non-innovative)  and  policy  guidance  provided  by  the  parent  agency,  and  to  study 
the  impact  of  organizational  mission,  culture,  and  practices  on  the  production  of  innovations.  The 
results  of  this  research  are  a  first  step  towards  developing  a  consistent,  quantitative,  and 
innovation-relevant  measure  of  research  laboratory  performance. 

Departing  from  the  citation-count  approach  to  valuing  patents,  this  study  instead  builds  on  an 
emerging  line  of  work  that  examines  relationships  between  the  accumulation  of  patents  within 
specific  technology  classes  over  time  and  the  innovativeness  of  the  filing  organizations.15  In 
particular,  this  study  is  indebted  to  the  conceptual  foundations  established  in  the  largely 
unpublished  work  of  Chris  Eusebi.  Eusebi  (2011)  analyzes  the  rate  and  sequence  of  patenting  in 
technology  classes  and  subclasses  to  determine  which  organizations  were  leading  patentees  early 
in  the  life  of  an  emerging  technology.  This  study  builds  on  Eusebi’s  idea  that  the  trends  in  patent 
filings  within  technology  classes  and  subclasses  over  time  contain  information  about  the 
innovativeness  of  the  inventors.  However,  rather  than  examining  trends  in  the  entire  patent 
record,  this  study  focuses  on  trends  by  technology  class  within  the  patent  portfolios  of  individual 
organizations. 

Not  including  the  absence  of  patents,  this  study  finds  four  distinct  patterns  of  patenting  that 
occur  within  single  organizations’  patent  portfolios  when  those  portfolios  are  examined  by 
technology  class  and  application  year.  At  least  two  of  these  patterns  can  be  logically  associated 
with  the  emergence  of  an  innovation.  Using  the  Department  of  Defense  (DOD)  in-house 
laboratories  as  case  studies,  this  dissertation  pursues  three  hypotheses  about  the  relationships 
between  these  patenting  patterns,  innovation,  and  evaluating  government  research  laboratories. 

The  first  hypothesis  driving  this  work  is  that  an  organization’s  patenting  patterns  reflect  its 
research  and  commercialization  strategies.  Each  of  the  basic  patenting  patterns  is  therefore  an 
indicator  of  a  particular  behavior  on  the  part  of  the  research  organization.  For  example,  when  an 
organization  produces  a  discovery  that  they  believe  is  novel  and  has  the  potential  to  be  widely 
used  there  is  a  corresponding  spike  in  their  patenting  behavior  reflecting  the  lab’s  attempt  to 
protect  the  core  invention  and  as  many  associated  technologies  as  possible.  Examining  in  detail 
the  technologies  in  which  these  patterns  occur  provides  insight  into  the  organization’s  research 
strategy  and  identifies  the  technology  areas  in  which  the  organization  thinks  it  can  be  influential. 

However,  an  organization’s  belief  in  the  innovative  usefulness  of  their  technology  will  not 
necessarily  be  realized.  There  are  a  number  of  reasons  why  a  technology  may  not  live  up  to  its 


15  The  terms  technology  classes  and  subclasses  refer  to  the  USPTO’s  technology  classification  system.  This  system 
is  explained  in  Chapter  2  and  Appendix  C. 
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inventor’s  expectations.  The  second  hypothesis  underlying  this  work  is  that,  regardless  of  a 
laboratory’s  own  evaluation  of  their  work,  the  relative  timing  of  an  organization’s  patent  filings 
compared  with  the  activity  of  other  inventors  in  the  same  field  can  be  an  independent  indicator  of 
innovativeness.  A  number  of  authors  have  observed  that  the  cumulative  number  of  patents  issued 
in  a  technology  area  often  follows  an  s-curve  (e.g.  see  Eusebi  &  Silberglitt  (N.d.),  Ernst  (1997)). 
Eusebi  (2011)  hypothesizes  that  organizations  that  have  patents  in  the  early  part  of  the  curve  are 
innovators  in  the  sense  that  they  were  leaders  in  a  technology  that  eventually  became  widely 
pursued.  Building  on  Eusebi ’s  approach  to  identifying  innovators  in  the  patent  record,  this 
dissertation  will  explore  the  ways  in  which  the  relative  timing  between  an  organization’s 
patenting  pattern  and  the  technology  class’s  overall  patent  s-curve  may  be  an  indicator  of 
innovativeness  on  the  part  of  that  organization. 

The  rate  at  which  an  organization  employs  certain  patenting  patterns  and  the  rate  at  which 
those  patents  lead  or  lag  the  rest  of  the  field  may  be  limited  by  the  organization’s  mission  and 
research  practices.  The  third  hypothesis  of  this  study  is  that  the  distribution  of  each  type  of 
patenting  pattern  in  an  organization’s  patent  portfolio,  the  technology  areas  in  which  they  occur, 
and  whether  that  pattern  occurs  early  or  late  in  the  overall  s-curve  are  all  related  to  the 
organization’s  mission,  research  and  commercialization  practices,  and  culture.  Statistical 
analyses  of  patenting  patterns  within  and  across  organizations  can  therefore  relate  organizational 
mission  and  culture  with  types  and  rates  of  innovation. 

1.3  Document  structure 

The  next  chapter  of  this  dissertation,  supported  by  Appendix  A  and  B,  continues  the 
introduction  by  providing  a  general  background.  Chapter  two  describes  in  more  detail  the 
difficulties  in  measuring  public-sector  research  innovativeness  and  explains  the  emerging 
approaches  to  analyzing  patent  trends  by  technology  class,  which  is  the  foundation  for  the 
methods  used  later  in  this  study.  Chapter  three  defines  the  common  patent  patterns  and  describes 
the  data  sources  and  methods  used  to  identify  these  patterns  in  organizations’  patent  portfolios. 
Chapter  four  develops  a  theory  of  the  research  activities  that  produce  each  pattern,  using 
example  cases  from  the  DOD  laboratories  to  illustrate  the  points,  and  explains  how  the  patenting 
patterns  can  be  indicators  of  innovation.  Chapter  five  presents  a  qualitative  and  quantitative 
example  of  using  the  overall  frequency  of  patenting  patterns  in  an  organization’s  portfolio  to 
study  the  missions,  cultures,  and  practices  that  contribute  to  research  innovation.  Chapter  six 
summarizes  this  study. 

1.4  Note  on  sources 

Many  individuals  throughout  the  DOD  laboratory  system  contributed  interviews,  background 
documents,  and  written  comments  for  this  study.  To  preserve  their  anonymity,  all  information 
obtained  from  these  sources  will  be  referenced  as  “Personal  Interview,  2012”. 
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Chapter  2  Background 


In  order  to  advance  the  study  of  factors  that  affect  the  production  of  innovations,  this 
dissertation  develops  several  new  patent-based  tools  that  are  suitable  for  monitoring  the 
production  of  innovations  by  public  research  organizations.  Following  the  broad  definition  of 
innovation  used  in  the  Oslo  Manual  (OECD,  2005)  and  early  patent  studies  (e.g.  Schmookler 
(1966)),  this  study  defines  an  innovation  as  any  novel,  non-obvious,  and  used  product.  This  study 
focuses  on  use  in  the  definition  of  innovation  because  the  transfer  and  use  of  research  products  is 
a  key  part  of  the  mission  of  government  research  organizations16  and  because  current  research 
evaluation  approaches  have  difficulty  measuring  the  actual  utility  of  research  products. 

Patents,  which  are  often  viewed  in  the  same  framework  as  journal  articles  as  a  count  of 
knowledge  outputs,  can  also  be  interpreted  as  indicators  of  inventions  that  are  released  to  the 
market  and,  hopefully,  used.  The  main  difficulty  in  interpreting  patents  in  this  manner  is 
confirming  actual  use.  While  there  are  a  variety  of  approaches  for  estimating  a  patent’s  value 
(summarized  in  Appendix  B,  note  that  value  is  not  necessarily  synonymous  with  use),  this  study 
draws  from  an  emerging  line  of  research  that  analyzes  patents  based  on  their  positions  along 
technology  maturation  s-curves. 

This  background  chapter  discusses  the  gaps  in  existing  research  evaluation  measures  (as 
applied  to  the  production  of  innovations  by  public  research  organizations),  why  use  is  a  key 
aspect  of  innovations,  and  how  location  along  the  technology  maturation  s-curve  can  be  a  proxy 
for  patent  value  and  use.  Section  2.1  reviews  the  many  approaches  that  have  been  used  to 
evaluate  research  organization  performance  and  the  challenges  of  finding  measures  that  are 
relevant  to  military  R&D  outcomes.  Section  2.2  defines  innovation  as  used  products  as  employed 
in  this  study.  Section  2.3  explains  the  occurrence  of  s-curves  in  the  patent  record  and  the 
potential  usefulness  of  patents  as  an  indicator  of  innovation.  These  sections  are  supplemented  by 
additional  literature  reviews  in  Appendix  A  and  Appendix  B. 

2.1  Overview  of  current  research  performance  measures 

When  evaluating  the  performance  of  a  research  organization,  there  are  at  least  a  dozen 
possible  approaches  an  analyst  can  choose.  While  the  most  appropriate  approach  will  logically 
depend  on  the  research  question,  the  type  of  organization  being  examined,  and  the  data  available, 
those  who  study  and  evaluate  research  organizations  are  relatively  free  to  select  their  own 
measures  “for  there  [is]  no  standard  set  of  methods  or  phenomena  that  every. . .  writer  [feels] 
forced  to  employ  and  explain,”  (Kuhn,  1962,  p.  13). 

16  The  mission  of  the  DOD  laboratories  to  “discover  new  phenomena  that  may  have  military  value”  (Coffey,  Lackie, 
&  Marshall,  2003),  implies  that  the  laboratories’  outputs  must  be  transferred  to  and  used  by  the  military. 
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The  list  of  possible  approaches  could  be  considerably  longer  than  a  dozen  if  the  question  is 
sufficiently  broad.  For  example,  while  considering  metrics  to  include  in  the  Technology  Value 
Pyramid  (TVP)  approach  to  R&D  management,  Industrial  Research  Institute  (IRI)  researchers 
uncovered  over  1 ,000  different  R&D  metrics  in  a  range  of  texts  going  back  400  years  to  when 
the  “princes  of  Europe  were  sponsoring  R&D. . .  [and]  asking  the  question:  Are  we  getting  our 
money's  worth?”  (Parish,  1998). 

Focusing  on  more  recent  applications,  Ruegg  &  Jordan  (2007)  identify  13  general 
approaches  to  R&D  evaluation  “that  have  proven  useful  to  R&D  program  managers  in  [U.S.] 
Federal  agencies,”  (Ruegg  &  Jordan,  2007,  p.  ii).  These  methods  can  be  used  independently  or  in 
combination. 

1 .  Peer  Review/Expert  Judgment  -  evaluating  research  programs  or  projects  based  on 
feedback  from  individuals  who  are  experts  in  that  area  of  study 

2.  Monitoring,  Data  Compilation,  and  Use  of  "Indicators"  -  on-going  tracking  of  inputs, 
outputs,  and  outcomes;  a  primarily  quantitative  variant  of  benchmarking  (#5) 

3.  Bibliometric  Methods:  Counts  and  Citation  Analysis  -  analyzing  numbers  of 
publications,  patents,  citations  they  receive,  and  networks  among  them 

4.  Bibliometric  Methods:  Data  Mining  -  analyzing  information,  beyond  counts  or  networks, 
contained  in  various  documents 

5.  Benchmarking  -  comparing  research  programs/projects  to  each  other  or  to  a  common 
standard 

6.  Technology  Commercialization  Tracking  Method  -  linking  commercialized  products  to 
contributing  research 

7.  Benefit-Cost  Case  Study  -  comparing  programs/projects  based  on  cost  and  quantified 
outcomes 

8.  Spillover  Analysis  -  estimating  the  public  and  private  value  of  program/project  outcomes 

9.  Network  Analysis  -  analyzing  relationships  among  researchers  based  on  a  variety  of 
records 

10.  Historical  Tracing  Method  -  tracking  the  historical  antecedents  to  or  the  ultimate  impacts 
of  a  research  discovery 

1 1 .  Econometric  Methods  -  using  mathematical  models  to  explore  potential  causal 
relationships  between  programs/projects,  their  characteristics,  and  their  outcomes 

12.  General  Case  Study  Methods  -  using  narratives  supported  by  other  data  to  explore 
program/project  processes  and  impacts 

13.  General  Survey  Methods  -  collecting  data  on  research  projects/programs  directly  from 
participants  or  other  stakeholders 

These  evaluation  approaches,  their  uses,  and  their  limitations  are  summarized  in  Appendix  A. 
A  good  metric  should  be  relevant  to  the  mission  of  the  organization,  credible,  “tolerably  easy  to 
calculate,”  and  understandable  (Parish,  1998).  While  each  approach  is  best  suited  to  certain  types 
of  questions,  and  no  one  approach  is  likely  to  be  appropriate  in  all  situations,  these  common 
approaches  are  not  well  suited  to  analyzing  the  performance,  particularly  the  innovative 
performance,  of  DOD  laboratories. 
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Consider,  for  example,  four  challenges  with  using  two  of  the  most  common  science- 
evaluation  tools,  bibliometric  measures  and  peer  review,  to  evaluate  the  innovativeness  of  public 
research  organizations  such  as  the  DOD  laboratories.  First,  bibliometric  measures,  when 
interpreted  as  measures  of  knowledge  generation,  are  not  necessarily  relevant  to  the  missions  of 
the  DOD  labs,  particularly  their  mission  to  transfer  discoveries  and  inventions  to  users.  Second, 
using  bibliometric  outputs  as  part  of  a  monitoring  scheme  for  knowledge  and  invention  transition 
results  in  an  incomplete  picture  because  many  transfers  occur  through  unobserved  interpersonal 
contacts.  Third,  current  approaches  to  valuing  patent  and  publication  counts  based  on  their  utility 
are  better  for  measuring  average  performance  than  detecting  innovative  performance.  Fourth, 
peer  reviews  of  organizational  process  or  outputs,  which  may  be  better  at  capturing  factors 
relevant  to  a  variety  of  missions,  are  inherently  subjective  and  different  sets  of  experts  can  and 
have  produced  contradictory  results  when  evaluating  the  same  organization.  The  following 
subsections  expand  on  these  four  issues. 

2. 1. 1  Relevance  of  bibliometrics  to  DOD  research  goals 

Bibliometric  measures  such  as  publication  and  citation  counts  are  good  performance 
measures  for  universities  whose  primary  mission  is  research  excellence.  However,  publication 
and  citation  counts  say  very  little  about  the  performance  of  industrial  or  government  research 
labs  whose  primary  mission  is  to  be  relevant  to  their  customers.  Consider  the  missions  for  DOD 
research  labs  spelled  out  by  the  Director  of  Defense  Research  and  Engineering  (DDR&E)  in 
1961  (Coffey  et  al.,  2003): 

1 .  discover  new  phenomena  that  may  have  military  value 

2.  communicate  the  needs  of  the  military  to  the  broader  scientific  and  technical  community 

3.  provide  objective  scientific  and  technical  advice  to  the  managers  of  research, 
development,  test,  and  engineering,  (RDT&E)  contracts 

4.  educate  military  planners  on  the  technical  art  of  the  possible 

5.  enhance  the  technical  training  of  military  officers  by  providing  laboratory  research 
experience 

An  evaluation  of  a  DOD  laboratory  that  uses  academic  publication  counts  and  impact  factors 
is  comparing  the  lab's  ability  to  generate  standard  science  outputs  to  that  of  other  organizations 
that  focus  on  academic  publication  outputs  (e.g.  research  intensive  universities).  But  do 
publication  counts  fully  evaluate  Brown's  first  mission  for  DOD  labs:  discover  new  phenomena 
that  may  have  military  value ?  One  could  argue  that  high  quality  science  will  naturally  yield 
military-relevant  discoveries,  but  this  assumption  should  be  explicitly  developed  in  any 
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assessment  that  relies  on  academic  bibliometric  measures. 
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Furthermore,  what  measure  tracks  whether  military  planners  have  been  sufficiently  educated  on  the  art  of  the 
possible? 
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2.1.2  Inputs,  outputs,  and  unobserved  knowledge  transition 

An  additional  challenge  for  evaluating  transition-focused  programs  based  on  their  inputs  or 
their  bibliometric  outputs  is  that  important  transition  activities  occur  without  a  written  record. 
The  issue  was  explored  in  detail  in  the  1975  National  Institute  of  Standards  and  Technology 
(NIST)  research  productivity  experiment  and  has  been  observed  in  several  historical  tracing 
studies  of  significant  innovations. 

In  1975,  NIST  (then  called  the  National  Bureau  of  Standards)  conducted  an  experiment  to 
determine  which  R&D  productivity  measures  were  appropriate  for  the  type  of  work  performed  at 
NIST  (J.  T.  Hall  &  Dixon,  1975).  The  study  made  a  critical  distinction  between  performance 
measures  and  productivity  measures.  Publication  counts  had  become  standard  individual  and 
institutional  performance  measures,  the  study  report  noted,  but  had  little  to  say  about 
productivity  if  productivity  was  defined  as  progress  toward  organizational  goals.  NIST's  primary 
research  goal  was  impact  through  the  transfer  of  knowledge  to  those  who  could  use  it.  The  NIST 
experiment  was  searching  for  a  tolerably  easy  way  to  evaluate  programs  based  on  knowledge- 
transfer  impacts  and  potentially  improve  their  performance. 

The  NIST  experiment  tracked  human  capital  and  financial  inputs,  conducted  detailed 
observations  of  work  processes,  and  tallied  outputs  including  publications,  devices,  and  software 
generated,  but  they  found  that  the  research  process  was  so  idiosyncratic  that  input  levels  were 
poor  predictors  of  output  levels.  Furthermore,  the  actual  rate  of  knowledge  transfer,  the  outcome 
of  interest,  was  unrelated  even  to  the  volume  of  outputs  (J.  T.  Hall  &  Dixon,  1975). 

A  major  measurement  challenge  for  NIST  was  that  knowledge  transfer  could  occur  without 
physical  outputs.  Other  studies  have  reported  similar  phenomena.  In  DOD's  Project  Hindsight, 
45%  of  science  knowledge  transfers  were  the  results  of  informal  personal  contacts.  In 
engineering  fields,  more  than  two-thirds  of  knowledge  transfers  tend  to  occur  through  personal 
contact  ("Technological  Forecasting  Lessons  from  Project  Hindsight"  cited  in  Hall  &  Dixon 
(1975)).  Allen  (1984)  observed  similar  results  in  his  study  of  communication  patterns  in 
government  research  contracts.  In  NSF's  TRACES  study,  informal  knowledge  transfer 
mechanisms  were  important  to  the  successful  development  of  9  out  of  10  societally  important 
technologies  (IIT  Research  Institute,  1968). 

2.1.3  Measuring  average  versus  innovative  performance 

Furthermore,  many  outputs  have  little  impact.  Both  patents  and  publications  have  highly 
skewed  distributions  when  their  value  is  measured  as  either  citations  or,  in  the  case  of  patents, 
commercial  value,  meaning  that  most  patents  are  found  in  the  low-value  end  of  the  scale  (B.  H. 
Hall,  Jaffe,  &  Trajtenberg,  2005;  Harhoff,  Narin,  Scherer,  &  Vopel,  1999;  Scherer  &  Harhoff, 
2000;  Trajtenberg,  1990).  The  value  distributions  are  so  skewed  that  value-weighted  bibliometric 
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statistics  are  more  likely  to  measure  a  lab's  ability  to  produce  average-value  outputs  than  its 

1  & 

ability  to  produce  high-value  outputs  (Scherer,  1965). 

2. 1.4  Consistency  of  peer  reviews  in  evaluating  research  processes 

Most  studies  of  government  labs  do  not  even  attempt  to  track  quantitative  performance 
measures,  instead  relying  on  expert  reviews  of  project  inputs,  processes,  and  outcomes.  Indeed, 
expert  reviews  are  the  standard  approach  to  evaluating  public  research  programs.  In  a  review  of 
evaluation  literature  for  the  Department  of  Energy,  Jordan  &  Malone  (2001)  note  that  the 
consensus  in  the  literature  is  that  "peer  review  is  the  most  effective  method  for  evaluating  the 
quality  and  value  of  fundamental  research,"  (p  11).  COSEPUP  (1999)  states,  "The  most  effective 
means  of  evaluating  federally  funded  research  programs  is  expert  review,"  (p  5).  In  addition, 
Harvey  A.  Averch  (2004)  argues  that  expert  reviews  are  the  best  way  to  assess  programs  that 
face  high  degrees  of  uncertainty,  particularly  when 

•  "A  public  agency  has  been  operating  a  "program"  for  a  number  of  years,  and  it  cannot  be 
certain  about  the  effective  quantity  or  quality  of  inputs  it  has  bought  during  those  years, 
and  there  is  no  clear  way  to  measure  these;" 

•  "The  expected  "benefits"  or  "outcomes"  of  the  program  are  highly  uncertain  in  the 
present  or  must  occur  in  the  future." 

•  "The  agency  does  not  know  with  precision  whether  decision-relevant  outcomes  can  be 
attributed  to  the  inputs  and  the  design  of  the  program."  (p  293) 

However,  if,  as  found  in  the  1975  NIST  study,  there  are  no  demonstrable  links  among 
research  inputs,  processes,  outputs,  and  mission-relevant  outcomes,  then  when  reviewers  draw 
conclusions  about  an  organization’s  performance  based  on  the  organization’s  inputs  or  processes, 
rather  than  by  considering  the  actual  outcomes,  they  are  basing  those  conclusions  on  their  own 
preferences  rather  than  on  evidence. 19  Not  only  can  such  reviews  produce  contradictory  results, 
but  there  is  also  no  guarantee  that  any  recommended  changes  to  the  organization  will  actually  be 
effective. 

Most  studies  of  the  DOD  labs  ask  panels  of  experts  to  judge  whether  descriptions  of  the  labs’ 
processes  and  inputs  match  the  experts’  expectations  for  a  high-quality  government  research 
laboratory.  While  these  studies  are  staffed  by  individuals  with  extensive  experience  managing 
research  programs,  the  bases  for  their  conclusions  can  be  unclear  and  different  panels  of  experts 
sometimes  report  contradictory  findings.20  For  example,  the  2009  JASONs  study  of  National 


18  Scherer  (1965)  actually  estimates  that  the  distribution  of  patent  values  does  not  have  a  finite  mean  or  variance. 
This  means  standard  statistical  inference  techniques  will  produce  biased  estimates  of  patent  values.  Taleb's  Black 
Swan  theory  provides  another  view  of  this  long-tailed  distribution  problem  with  an  emphasis  on  the  unpredictability 
of  such  extreme  outliers  (Taleb,  2010). 

19 

For  example,  G.  B.  Jordan,  Streit,  &  Binkley  (2000)  find  that  researchers  employed  in  different  types  of  work 
(e.g.  more  basic  or  more  applied)  prefer  different  types  of  work  environments. 

20 

Recent  expert  studies  of  the  DOD  labs  include  the  Defense  Science  Board  study  of  DOD  Basic  Research 
(Defense  Science  Board,  2012),  the  JASONs  study  of  the  same  (JASONs,  2009),  the  National  Defense  University 
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Security  S&T  (JASONs,  2009)  found  that  DOD  basic  research  was  becoming  too  tied  to  short¬ 
term  applications.  However,  the  2012  Defense  Science  Board  (DSB)  study  of  Defense  Basic 
Research  (Defense  Science  Board,  2012)  disagreed,  finding  that  DOD  basic  research  was 
appropriately  targeted  and  in  good  health.  Neither  study  directly  addresses  whether  the  defense 
basic  research  program  is  helping  to  meet  warfighter  needs. 

2.2  A  DOD-relevant  definition  of  innovation  as  use 

In  order  to  focus  this  review  on  laboratory  outputs  and  outcomes  that  are  relevant  to 
warfighter  needs,  this  study  employs  a  definition  of  innovation  based  on  use.  After  all,  the  DOD 
laboratory  mission  to  “discover  new  phenomena  that  may  have  military  value”  (Coffey  et  al., 
2003),  implies  that  the  laboratories’  outputs  are  transferred  to  and  used  by  the  military. 

Borrowing  from  Schmookler  (1966),  this  study  defines  an  innovation  as  a  concrete  product 
that  is  widely  used,  novel,  and  non-obvious.  This  definition  provides  guidelines  to  separate  the 
non-innovative  -  an  unimplemented  idea,  a  product  with  promise  but  few  users  (e.g.  the  flying 
car)  -  from  the  innovative  -  the  iPhone  and  its  user  interface,  stealth  fighter/bombers. 

General  inventions,  as  opposed  to  innovations,  fail  one  of  the  three  criteria;  they  are  either 
not  widely  used,  not  novel,  or  are  obvious.  For  example,  Microsoft  developed  and  sold  the 
Tablet  PC  in  the  early  2000s  (Fontana,  2010),  but  it  has  not  become  as  widely  used  as  the  Apple 
iPad.  Android  tablets  have  captured  significant  market  share,22  but  patent  infringement  lawsuits 
(e.g.  Apple  v.  Samsung  (Lowensohn,  2012))  have  held  that  they  are  derivatives  of  the  iPad  (i.e. 
Android  tablets  are  less  novel  and  therefore  less  innovative  than  the  iPad). 

Some  may  object  that  this  definition  of  innovation  conflates  research  and  development, 
manufacturing,  marketing,  and  the  fickleness  of  the  market.  That  is  actually  the  point.  The  most 
innovative  organizations,  according  to  this  definition,  are  those  that  understand  what  their 
customers  need  and  want  and  execute  their  research,  development,  and  manufacturing  functions 
to  deliver  it.23 

In  the  context  of  the  DOD  labs,  this  study’s  definition  of  innovation  focuses  on  the  labs’ 
ability  to  both  discover  new  phenomena  that  may  have  military  value  and  also  their  ability  to 


series  by  former  laboratory  executives  (Chait,  2009;  Lyons,  2012;  Lyons  &  Chait,  2009;  Lyons,  Chait,  &  Willcox, 
2009;  Lyons  et  al.,  2005),  and  the  2010  Decker  panel  on  improving  Army  Basic  Research  (Decker  et  al.,  2010). 

21 

Other  analysts  have  identified  issues  of  reviewer  bias  in  other  settings.  G.  Jordan  &  Malone  (2001)  observe  that 
reviewers  may  display  favoritism  to  certain  researchers  or  lines  of  research  or  respond  negatively  to  research  that 
contradicts  their  own  work.  Bias  can  also  arise  from  the  information  a  program’s  staff  chooses  to  reveal.  Averch 
(2004)  emphasizes  that  reviewers  should  carefully  judge  the  credibility  of  an  organization’s  claims  and  be 
particularly  skeptical  when  failures  are  blamed  on  shortages  of  money  or  personnel. 

22 

For  market  shares  of  Android,  Apple  (iOS),  and  Microsoft  devices  see  IDC  (2013). 

23 

As  Peter  Drucker  wrote,  inventions  without  a  focus  on  need  and  users  are  like  Da  Vinci’s  drawings  -  brilliant 
ideas  that  are  never  adopted;  inventors  who  do  not  systematically  consider  customer  requirements  miss  opportunities 
or  misinterpret  the  need  (Drucker,  1985). 
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connect  with  program  officers  and  contractors  who  can  bring  these  innovations  to  the  field. 
However,  identifying  use  can  be  particularly  challenging  for  the  products  of  defense  research 

24 

since  the  idea  must  pass  through  many  different  organizations  on  its  way  from  bench  to  field, 
and  the  ultimate  use  may  be  classified. 

2.3  Patent  trends  as  an  indicator  of  innovation 

Admittedly,  deciding  when  an  invention  crosses  over  into  innovative  territory,  as  defined 
above,  is  a  subjective  judgment.  In  theory,  patent  data  can  help  make  this  decision  more 
objective.25  To  receive  a  patent,  according  to  U.S.  patent  law,  an  invention  must  be  new,  non- 
obvious,  and  potentially  useful.  The  definitions  of  these  terms  are  laid  out  in  the  law,  answering 
the  question  of  how  new,  useful,  and  non-obvious  an  invention  must  be  to  qualify  as  an 
innovation.  Specifically,  an  invention  must  not  be  known  in  the  public  domain  more  than  a  year 
before  the  filing  date,  must  be  implementable,  and  must  be  not  obvious  “to  one  of  ordinary  skill 
in  the  art.”26  However,  there  are  certainly  trivial  patents  that  do  not  meet  the  connotation  of 
innovation.  The  distribution  of  patent  values  is  highly  skewed  (Scherer,  1965);  most  patents  are 
commercially  worthless,  some  are  even  ‘wacky’  (Czamitzki,  Hussinger,  &  Schneider,  2011),  and 
only  a  few  provide  significant  returns  to  their  owners. 

As  described  in  Appendix  B,  several  approaches  have  been  developed  to  measure  the  value 
(for  different  definitions  of  value)  of  any  given  patent  and  distinguish  the  significant  patents 
from  the  trivial.  For  example,  the  number  of  citations  received  by  a  patent  from  other  later 
patents  (i.e.  forward  citations)  has  been  used  as  a  proxy  for  a  patent’s  commercial  value  (which 
is  concept  similar  to  degree  of  use)  or  as  a  proxy  for  a  patent’s  originality  (i.e.  novelty).  Like  the 
research  evaluation  methods  described  in  Appendix  A,  these  measures  of  patent  value  have  their 
uses  and  their  limitations.  However,  thinking  about  patents  in  terms  of  a  series  of  actions  rather 
than  as  discrete  and  independent  points  can  yield  additional  insights  into  their  relative  value  and 
the  organizational  processes  that  led  to  their  creation. 

The  measures  developed  in  this  study  start  from  Eusebi’s  insight  that  the  accumulation  of 
patents  in  specific  technology  classes  over  time  is  the  result  of  many  organizations  making  bets 
on  the  importance  of  future  technologies  (Eusebi  &  Silberglitt,  n.d.).  Collectively,  these  bets 
and  the  response  of  the  patent  office  to  a  technology’s  popularity  result  in  an  s-curve-like  growth 


24 

as  opposed  to  an  industrial  research  lab  which  may  be  more  closely  connected  to  its  related  manufacturing  and 
marketing  departments 

25 

Patenting  is  not  the  only  way  in  which  DOD  labs  transfer  technologies  to  users.  It  may  not  even  be  their  primary 
method  of  technology  transfer,  but  patents  may  be  representative  of  a  lab’s  general  research  focus.  This  issue  of 
patents  and  representativeness  is  discussed  in  section  3.3  and  in  section  3.7. 

26 

The  patent  examiner  being  the  person  whose  skill  in  the  art  matters  the  most.  For  more  information  on  patent  law, 
see  the  Bitlaw  website  which  provides  non-technical  summaries  with  links  to  official  sources. 

27 

See  Appendix  C  for  an  overview  of  the  USPTO  technology  class  system. 
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of  patents  that  corresponds  with  the  maturation  of  that  technology  over  time.  As  the  following 
sections  explain,  patents  that  are  filed  near  the  take-off  inflection  point  of  a  technology  s-curve 
are  thought  to  have  a  larger  commercial  potential,  and  to  generally  be  more  “innovative”,  than 
those  filed  later  (Eusebi,  2011).  This  study  expands  this  line  of  work  by  developing  hypotheses 
about  the  meaning  of  analogous  patterns  within  single  organization  patent  portfolios,  and 
comparing  those  single-organization  curves  to  the  s-curve  trends  in  the  field  as  a  whole. 

2.3. 1  S-curves  in  the  cumulative  growth  of  patents  per  technology  class 

The  cumulative  number  of  patents  assigned  to  a  technology  class  over  time  often  follows  an 
s-shaped  curve.  Technology  class  here  referrers  to  the  U.S.  Patent  Classification  System  codes. 
The  U.S.  Patent  Classification  System  is  a  hierarchical  system  of  classes  and  subclasses  (usually 
designated  using  numeric  codes  separated  by  a  slash  in  the  form  “class  number/subclass 
number”)  used  by  the  U.S.  Patent  Office  to  identify  similar  patents  and  patent  applications. 

Every  patent  is  assigned  to  one  primary  and  any  number  of  additional  (called  cross  reference) 
class/subclasses.  Together,  these  assigned  class/subclasses  fully  describe  the  technology  and 
applications  areas  covered  by  the  patent’s  claims  (see  Appendix  C  for  more  details).  The 
cumulative  number  of  patents  assigned  to  a  given  class/subclass  often  begins  to  resemble  an  s- 
curve  over  time. 

“S-curve”  is  the  generic  term  for  any  of  several  of  functions  (e.g.  the  logistic  function, 
Gompertz  function,  or  many  cumulative  probability  distribution  functions)  that,  when  graphed, 
look  like  tilted  S’es  (Figure  2.1).  The  curve  starts  flat,  and  then  at  some  point  -  the  point  of  “take 
off’  -  the  slope  of  the  curve  begins  increasing  at  an  increasing  rate  (positive  second  derivative). 
Next,  at  the  inflection  point,  the  slope  begins  to  decline  at  an  increasing  rate  (negative  second 
derivative)  until  it  becomes  flat  again  as  it  approaches  the  curve’s  maximum  value. 


28 

“[A  patent’s]  claims  state,  in  technical  and  precise  terms,  the  subject  matter  of  the  invention  (or  discovery),  as 
well  as  its  purpose,  principal  properties,  how  it  operates  and  the  methods  it  employs,”  (Strumsky  et  ah,  2012,  p.  4). 
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Figure  2.1  Generic  s-curve 


Emerging  Growth  Maturity  Saturation 


This  pattern  can  be  described  in  terms  of  regions.  Different  authors  use  different  names  for 
the  various  regions.  Some  authors  identify  six  regions,  including  birth  (at  the  far  left)  and  death 
(at  the  far  right).  This  study  borrows  a  four-region  terminology  from  Chen,  Chen,  &  Lee  (2010) 
and  Ernst  (1997).  The  first  region,  prior  to  takeoff,  is  called  the  Emerging  region.  The  period  of 
increasing  growth  (positive  second  derivative)  is  the  Growth  region.  The  period  of  decreasing 
growth  (negative  second  derivative)  is  the  Maturity  region.  The  final  period,  as  the  curve  flattens 
toward  its  maximum  value,  is  called  the  Saturation  region. 

Figure  2.2  shows  the  cumulative  number  of  patents  filed  (top  line)  in  Vehicle  Navigation 
Using  Global  Positioning  Systems  (GPS)  (class  701/213)  between  1985  and  2010.  Class  701/213 
was  selected  for  this  illustration  because  it  is  a  particularly  clean  case  of  an  s-curve  in  the  patent 
record.  This  example  first  appeared  in  Eusebi  (201 1). 

The  fdled  series  in  the  lower  portion  of  Figure  2.2  shows  the  number  of  new  applications 
(which  ultimately  resulted  in  issued  patents)  each  calendar  year.  The  top  series,  showing  the 
cumulative  number  of  patents,  has  an  s-curve  shape.  Emerging,  Growth,  Maturity,  and  Saturation 
regions  are  also  approximately  labeled  in  Figure  2.2.  The  number  of  patents  fded  each  year 
begins  increasing  around  1994,  the  year  that  GPS  code  was  released  to  the  general  public. 
Presumably  as  a  result  of  this  newly  available  technology,  the  cumulative  number  of  patents  filed 

14 


“takes  off’;  its  slope  begins  increasing  at  an  increasing  rate.  By  the  mid-2000s,  this  technology 
area  reaches  its  inflection  point,  and  fewer  new  patents  are  granted.  The  cumulative  sum  line 
begins  to  flatten,  and  by  about  2006  has  leveled  off  at  a  maximum  of  almost  2000  patents  total 
fded  in  this  technology  area. 

Figure  2.2  Illustrating  patent  s-curves  in  a  sample  technology  class/subclass 
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■Cumulative  Issued 


Source:  Analysis  of  U.S.  Patent  and  Trademark  Office  (USPTO)  2009  patent  grant  bibliographic  data  file 

Analyzing  the  cumulative  number  of  patents  granted  in  each  class/subclass  in  the  USPTO 
system  shows  that  this  s-curve-like  growth  is  common,  although  there  are  exceptions.  Some 
classes  seem  to  have  multiple  growth  phases  resulting  in  cumulative  sum  curves  that  look  like 


29 

Early  exploratory  work  for  this  study  used  statistical  analysis  software  to  fit  a  logistic  equation  to  every 
class/subclass  cumulative  sum  series  in  the  USPTO  2009  patent  grant  bibliographic  data  file.  Of  the  152,128 
class/subclasses  that  appeared  at  least  once  in  the  data  file,  this  method  obtained  parameter  estimates  for  106,903 
class/subclass  trends  (70%);  45,225  class/subclass  cumulative  sum  curves  had  a  shape  too  different  from  the  logistic 
curve  or  had  too  few  observations  to  obtain  parameter  estimates  using  R’s  “nls”  package.  This  exercise  should  not 
be  taken  to  mean  that  the  cumulative  number  of  patents  in  a  technology  class  necessarily  follows  a  simple  logistic 
curve.  There  are  a  number  of  other  possible  functional  forms.  See  Modis  (2007)  for  cautions  regarding  attempting  to 
fit  s-curve-like  data  to  a  logistic  function. 
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multiple  “loglet”  curves  (Meyer,  Yung,  &  Ausubel,  1999).  For  example,  multiple  wavelength 
laser  arrays  (see  left  side  of  Figure  2.3)  seem  to  have  grown  in  two  phases.  The  first  burst  of 
work  occurred  in  the  mid-1980s;  in  the  1990s,  there  was  very  little  activity  in  this  field;  and  in 
the  2000s  patenting  in  the  class  took  off  again.  Other  classes  have  either  yet  to  take  off  or 
reached  maturity  before  the  start  of  the  data  file  and  therefore  have  flat  cumulative  sum  curves 
(see  right  side  of  Figure  2.3).  In  the  second  example,  there  was  some  activity  in  solar-pumped 
lasers  in  the  1970s,  at  the  very  beginning  of  the  dataset,  but  only  three  patents  have  been  filed  in 
the  class  since  1980. 

Figure  2.3  Non-s-curve  technology  class  examples:  multiple  “loglets”  (left)  and  flat  (right) 
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Source:  Analysis  of  USPTO  2009  patent  grant  bibliographic  data  file 

Why  does  the  growth  in  patents  per  class  follow  an  s-curve? 

S-curves  describe  growth  in  a  self-limiting  population  (Kucharavy  &  De  Guio,  2007;  Modis, 
2007).  The  population  of  a  species  competing  for  resources,  either  with  another  species  (e.g. 
predatory-prey)  or  with  its  environment,  follows  an  s-curve  (Modis,  2003,  201 1).  The  market 
shares  of  competing  technologies  can  be  modeled  using  these  s-curve  population  dynamic 
models  (Dattee,  2007;  Meyer  et  al.,  1999;  Pistorius  &  Utterback,  1997).  The  diffusion  of  ideas  or 
the  spread  of  disease  from  the  early  adopters  to  the  rest  of  the  susceptible  population  also  follows 
an  s-curve  (Bettencourt,  Cintron- Arias,  Kaiser,  &  Castillo-Chavez,  2006;  Rogers,  1995),  as  do 
improvements  in  the  performance  of  a  class  of  technologies  (Asthana,  1995;  Christensen,  1997; 
Foster,  1986). 

The  population  of  patents  in  a  technology  class  is  limited  by  two  factors:  inventors’ 
collective  productivity  in  the  field  and  patent  examiners’  collective  willingness  to  accept  the 
novelty  of  a  new  claim.  At  first  inventors  produce  few  patentable  discoveries  in  the  field  and  few 
patents  exist  in  the  class.  At  some  point  a  technology  area  begins  to  "take  off'  and  increasing 
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The  USPTO  2009  patent  grant  bibliographic  data  file  which  contains  patents  granted  since  1976. 


16 


numbers  of  patents  are  applied  for  and  awarded  in  that  class.  As  the  technology  area  becomes 
mature,  three  complementary  forces  slow  the  rate  of  growth  in  patents:  one,  additional  patent 
applications  contain  fewer  and  fewer  things  that  are  sufficiently  novel  to  merit  a  patent,  and 
examiners  accept  fewer  patent  applications  in  that  class  (Eusebi,  2012);  two,  diminishing  returns 
to  research  reduce  the  number  of  patentable  discoveries  in  that  field  that  can  be  easily 
produced;  and  three,  less  successful  firms  exit  the  market  (see  cases  studies  in  Foster  (1986) 
and  Christensen  (1997))  reducing  the  number  of  inventors  still  searching  for  additional 
breakthroughs. 

2.3.2  Location  on  the  S-curve  as  an  indicator  of  innovativeness 

The  business  strategy  literature  suggests  that  organizations  inventing  in  the  growth  part  of  the 
technology  s-curve  are  more  innovative  than  others  in  the  sense  that  they  are  creating  novel 
technologies  that  are  likely  to  be  used  in  the  marketplace.  More  specifically,  one  stream  of  the 
business  literature  recommends  a  company  carefully  analyze  the  rate  at  which  a  technology  is 
improving  (whether  it  is  in  the  growth  or  maturity  phase)  when  deciding  whether  to  invest  in 
incremental  improvements,  invest  in  a  totally  new  technology  direction,  or  acquire  a  possible 
competitor.  If  the  technology  is  still  growing,  the  company  may  want  to  invest  in  improvements 
itself  or  acquire  a  competitor  that  is  threatening  to  introduce  an  incremental  improvement.  If  the 
technology  is  mature,  then  it  is  better  to  invest  in  new  directions  and  watch  out  for  disruptive 
competitors  that  are  operating  on  the  growth  part  of  a  new  s-curve.  This  kind  of  tracing  can  be 
done  by  carefully  examining  the  performance  characteristics  of  technologies,  as  in  Christensen 
(1997),  Asthana  (1995),  or  Foster  (1986),  or  by  tracking  the  accumulation  of  patents  within 
technology  classes  as  demonstrated  in  Eusebi  &  Silberglitt  (N.d.),  Chen  et  al.  (2010),  Andersen 
(1999),  or  Ernst  (1997). 

Turning  this  recommendation  around,  because  they  represent  the  work  of  someone  who 
moved  on  a  profitable  technology  area  before  it  was  obviously  "hot",  patents  that  are  filed  before 
the  technology  takes  off  are  more  innovative  than  ones  that  are  filed  when  the  technology  area  is 
mature.  Early  patentees  are  leaders  in  a  field  that  many  other  inventors  later  recognized  as  an 
important  area  for  investment.  Those  early  patents  were  novel  and,  in  a  general  sense,  were 
eventually  widely  used  as  others  built  on  or  imitated  them. 

However,  it  is  possible  to  be  too  early  as  well  as  too  late.32  Patents  filed  in  the  far  left  tail  of 
the  curve  may  just  be  “wacky”  -  trivial  or  useless  combinations  of  distantly  related  technologies 
(Czamitzki  et  al.,  2011).  Even  if  they  represent  useful  combinations  of  technologies,  early 
patents  may  be  “before  their  time.”  For  example,  the  patent  may  include  technologies  that  cannot 

31 

Foster  (1986)  describes  several  cases  in  which  an  industry  was  reaching  the  physical  limits  of  a  technology  (e.g. 
the  theoretical  maximum  chemical  yield  or  the  maximum  strength  of  a  material)  yet  continued  to  spend  significant 
sums  on  research  with  very  little  resulting  improvements  in  performance. 

32 

“It’s  always  about  timing.  If  it’s  too  soon,  no  one  understands.  If  it’s  too  late,  everyone’s  forgotten.”  -  Anna 
Wintour  (Cordero,  201 1). 
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yet  be  manufactured  at  an  affordable  rate.  Alternately,  the  early  market  for  the  technology  may 
be  small  compared  to  its  later  size  limiting  the  early  technology’s  adoption  and  other  inventors’ 
awareness  of  it. 

Therefore  the  most  innovative  patents  in  a  technology  class  are  probably  among  those  that 

33 

are  filed  near  the  point  of  takeoff  in  the  late-emerging  or  early-growth  parts  of  the  s-curve. 

These  patents  represent  research  done  around  the  time  inventors  figured  out  how  to  make  the 
field  a  success  and  before  the  technology  became  mature  and  diminishing  returns  to  research 
resulted  in  only  incremental  discoveries.  Following  this  argument,  the  most  innovative  research 
organizations  are  those  that  tend  to  have  patents  in  the  late-emerging  and  early-growth  parts  of 
technology  classes  with  s-curves. 

S-curve  example  in  military  technology 

This  theory  seems  to  be  valid  for  military  as  well  as  commercial  technologies.  For  example, 
patent  data  show  an  increase  in  stealth  technology  activity  in  the  early  1980s.  Figure  2.4  shows 
the  rate  of  patenting  in  a  stealth  technology-related  class  (342/2  Radio  Wave  Absorber  for 
Aircraft  or  Missiles).  The  bottom,  blue  series  shows  the  number  of  patents  filed  (and  ultimately 
issued)  in  this  class  by  application  year  while  the  top,  red  series  shows  the  cumulative  number  of 
patents  in  the  class  by  application  year.  While  it  is  unclear  if  this  class  has  reached  maturity 
(patents  filed  in  the  late  2000s  are  probably  still  being  processed  and  do  not  appear  in  the 
dataset),  the  rate  of  patenting  in  this  class  increased  around  1984.  Black  lines  show  the 
approximate  slope  of  the  cumulative  sum  series  before  and  after  1984.  Given  that  it  takes  some 
time  to  progress  from  the  start  of  a  research  project  to  a  patent  application,  research  that 
contributed  to  this  increase  in  patenting  likely  occurred  sometime  in  the  late- 1970s  or  early- 
1980s.34 

A  patent  representing  a  significant  innovation  in  stealth  technology  appears  approximately 
where  one  would  expect  if  the  hypothesis  that  big  innovations  appear  just  before  the  takeoff 
point  is  correct.  Stealth  technology  was  revolutionized  in  the  mid-1970s  when  defense 
contractors  Lockheed  and  Northrop  independently  developed  methods  to  reduce  the  radar  cross 

'J  ^ 

section  of  aircraft  by  several  orders  of  magnitude  (Aronstein  &  Piccirillo,  1997).  Lockheed 
patented  their  approach  in  1979  (U.S.  patent  5, 250, 950), 36  and  went  on  to  build  the  F-l  17A,  the 
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Independent  co-discoveries  and  delays  between  conducting  research  and  filing  a  patent  may  lead  to  some 
innovative  patents  appearing  just  after  the  point  of  takeoff. 

34 

While  one  could  trace  the  conceptual  ancestors  of  any  particular  invention  arbitrarily  far  back  in  time,  studies  of 
private-sector  projects  find  that  patents  tend  to  be  filed  early  in  the  R&D  process  (Griliches,  1990,  p.  1674). 

35 

Radar  cross-section  is  a  measure  of  how  visible  an  object  is  to  radar  sensors.  Objects  with  smaller  cross-sections 
are  harder  to  detect.  A  number  of  factors  can  affect  radar  cross-section  including  the  shape  of  the  object,  the  surface 
material  of  the  object,  and  the  angle  between  the  sensor  and  the  object. 

36  Northrop  seems  not  to  have  filed  a  utility  patent  on  their  stealth  technology,  although  they  have  several  design 
patents  for  military  aircraft.  This  may  have  something  to  do  with  the  different  contract  terms  Lockheed  negotiated 
during  the  stealth  fighter  demonstration  project.  Lockheed  entered  the  stealth  design  contest  on  a  non-standard 
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world’s  first  stealth  fighter,  which  became  operational,  but  still  classified,  in  1983  (Rich  &  Janos, 
1996).37 

Figure  2.4  Illustrating  the  location  of  innovative  patents  in  the  s-curve 
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Source:  Analysis  of  USPTO  2009  patent  grant  bibliographic  data  file 

It  is  difficult  to  say  whether  Lockheed’s  work  on  the  F-l  17A  was  the  breakthrough  that  made 
later  work  possible,  or  if  this  type  of  stealth  was  being  simultaneously  pursued  by  many 
inventors  and  Lockheed  was  just  the  first  of  many  to  make  it  work.  The  challenge  of  responding 
to  increasingly  sophisticated  air  defense  systems  had  been  an  issue  throughout  the  Cold  War.  For 
example,  in  the  late- 1950s  Lockheed  developed  radar  absorbing  coatings  for  the  SR-71  in 
response  to  President  Eisenhower’s  request  for  the  lowest  possible  radar  cross  section  (Rich  & 
Janos,  1996,  pp.  197-199).  The  expansion  in  stealth-related  work  in  the  1980s  could  simply  have 
been  due  to  increasing  concerns  on  the  part  of  military  planners.  However,  before  Lockheed  and 
Northrop’s  work  in  1975,  no  one  knew  how  to  actually  calculate  the  radar  cross  section  of  a  new 
aircraft  design  (Aronstein  &  Piccirillo,  1997),  and  in  the  1980s,  with  that  problem  solved, 
coincidentally  or  not,  stealth  went  from  being  a  niche  business  to  an  expanding  one.  Either  way, 


contract  that  allowed  the  company  to  keep  all  of  their  intellectual  property  (Rich  &  Janos,  1996).  Northrop 
developed  their  equivalent  technology  under  contract  to  the  Air  Force  (Aronstein  &  Piccirillo,  1997). 

37 

Lockheed’s  stealth  vehicle  patent  was  not  issued  until  1993.  The  14-year  delay  probably  reflects  a  secrecy  order 
on  the  patent  since  the  existence  of  the  F-l  17A  was  not  officially  acknowledged  until  1988  (Morrocco,  1988). 
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based  on  the  location  of  their  stealth  patents  in  the  class  s-curve,  Lockheed  should  be  called  an 
innovator  in  stealth  technology. 

2.4  From  cumulative  s-curves  to  within-organization  patenting  patterns 

The  region  -  emerging,  growth,  maturity,  or  saturation  -  in  which  a  given  patent  appears  can 
be  an  indicator  of  the  innovativeness  and  usefulness  of  the  underlying  idea.  As  demonstrated 
with  the  Lockheed  stealth  example  above,  patents  that  appear  in  the  late-emerging  or  early- 
growth  phase  of  the  cumulative  patent  s-curve  can  be  indicators  of  this  type  of  innovative 
leadership.38  In  studies  of  drivers  of  innovation,  the  number  of  leading  patents  an  organization 
has  could  be  used  as  the  objective  measure  of  innovativeness  (i.e.  the  dependent  variable)  and 
compared  with  various  factors  (i.e.  independent  variables)  such  as  the  qualifications  of  research 
staff  or  organizational  mission  and  culture. 

However,  there  is  additional  information  in  organizations’  patterns  of  patenting  that  may  help 
focus  the  study  of  innovation.  Just  as  the  cumulative  behaviors  of  organizations  result  in  the 
accumulation  of  patents  along  an  s-curve,  the  cumulative  behavior  of  inventors  within  a  single 
organization  creates  a  coherent  patenting  trend.  Sometimes  this  within-organization  patenting 
trend  resembles  an  s-curve,  but  other  times  it  is  more  irregular  and  spikey.  Furthermore,  just  as 
patent  s-curves  occur  because  many  organizations  are  betting  that  the  technology  will  be 
profitable  (Eusebi  &  Silberglitt,  n.d.),  patentees  within  an  organization  are  responding  to  some 
signal  about  the  importance  of  their  research  areas.  Which  signals  correspond  with  which  within- 
organization  patenting  patterns  is  the  subject  of  the  next  two  chapters.  Understanding  the  drivers 
of  these  within-organization  patenting  patterns  may  lead  to  a  better  understanding  of  how 
different  organizations  pursue  invention  and  innovation.  Section  4.2  will  return  the  discussion  to 
innovation  and  the  cumulative  field  s-curve  by  comparing  the  within-organization  patenting 
patterns  with  contemporaneous  trends  in  the  field  as  a  whole. 


38 


A  second  example,  IBM  in  nanotechnology,  is  discussed  in  4.2. 
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Chapter  3  Within-Organization  Patent  Patterns:  identification  and 
data  sources 


Just  as  patents  filed  over  time  by  multiple  inventors  and  organizations  can  produce  an  s-curve 
in  the  full  patent  database,  the  sequences  of  patents  filed  over  time  by  inventors  employed  by  any 
single  organization  can  form  distinct  patterns.  As  discussed  in  Section  2.3,  s-curves  in  the  full 
patent  record  have  been  interpreted  as  indicators  of  technology  diffusion  (Ernst,  1997)  and  as 
indicators  for  the  occurrence  of  (Eusebi,  201 1)  or  remaining  potential  for  (Y.  H.  Chen  et  al., 
2010)  innovation  in  a  field  of  technology.  There  is  as  yet  no  equivalent  set  of  theories  explaining 
how  within-organization  patenting  patterns  occur,  what  the  specific  patterns  signify,  or  why 
different  patterns  occur  with  different  rates  in  different  organizations.  This  dissertation  aims  to 
fill  this  gap. 

This  study  has  three  hypothesis  regarding  the  causes  and  implications  of  within-organization 
patenting  patterns.  First,  an  organization’s  pattern  of  patenting  in  a  technology  class  over  time 
reflects  the  research  and  commercialization  approach  the  organization  has  applied  to  that 
technology.  Second,  certain  patterns  themselves  as  well  as  the  location  of  that  pattern  on  the 
broader  technology  class  s-curve  can  be  indicators  of  innovation.  This  study  hypothesizes  that 
organizations  produce  distinctly  different  patenting  patterns  when  they  believe  a  line  of  work  is 
highly  valuable  and  potentially  innovative  than  when  they  believe  a  line  of  work  is  of  average 
value.  In  addition,  since  an  organization’s  individual  patenting  pattern  is  part  of  the  larger 
technology  class  patenting  trend,  within-organization  patenting  patterns  that  occur  early  in  the 
larger  technology  class  s-curve  can  be  an  indicator  of  actual  innovation  as  described  in  section 
2.3.2.  Third,  the  distribution  of  patenting  patterns  across  various  technologies  in  an 
organization’s  portfolio  is  related  to  the  organization’s  research  mission,  culture,  and  procedures. 
Research  organizations  with  different  missions,  cultures,  and  procedures  will  therefore  have 
different  distributions  of  patterns  in  their  patent  portfolios. 

This  study  explores  these  hypotheses  in  three  steps.  First,  this  chapter  describes  the  patterns 
that  occur  within  laboratory  patent  portfolios  and  demonstrates  their  occurrence  within  a  set  of 
test  cases:  the  DOD  laboratories.  Second,  chapter  4  addresses  the  second  hypothesis  by 
developing  the  theoretical  argument  for  the  link  between  innovation  (as  defined  as  a  concrete 
product  that  is  widely  used,  novel,  and  non-obvious)  and  these  patenting  patterns,  again  drawing 
on  results  from  the  DOD  laboratories  for  supporting  or  contradictory  evidence.  Third,  chapter  5 
demonstrates  approaches  for  analyzing  relationships  between  patenting  patterns  within  the  DOD 
laboratory  portfolios  and  characteristics  of  the  laboratories  such  as  technological  focus  area, 
budget,  and  character  of  work  (e.g.  more  basic  or  more  applied,  more  investigator-directed  or 
more  service-directed). 
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3.1  Identifying  within-organization  patent  patterns:  methodology  overview 

This  study  finds  that  histograms  of  a  single  organization’s  patents  constructed  along 
technology  class  and  application  year  (e.g.  a  bar  graph  of  the  number  of  patents  per  application 
year  filed  by  a  given  organization  in  a  given  technology  class)  can  be  classified  into  four  pattern 
types.  The  frequency  with  which  each  pattern  appears  in  an  organization’s  portfolio  may  be 
related  to  the  organization’s  research  mission,  culture,  procedures,  and  innovativeness.  If  the 
definitions  of  these  patterns  and  the  behaviors  behind  them  proposed  in  this  study  hold,  then 
these  patterns  can  be  used  to  both  evaluate  the  alignment  between  a  research  organization’s 
stated  mission  and  its  actual  performance  as  well  as  to  identify  specific  innovative  outputs. 

This  chapter  explains  this  study’s  methodology  for  identifying  within-organization  patenting 
patterns  in  5  steps.  First,  section  3.2  describes  the  organizations  -  the  DOD  laboratories  -  whose 
data  have  been  used  to  develop  the  methods  in  this  dissertation.  Second,  section  3.3  describes  the 
sources  of  DOD  patent  data  used  in  this  study.  Third,  section  3.4  provides  summary  statistics  for 
this  data  set  and  describes  the  limitations  of  the  data  sources.  Fourth,  section  3.5  presents  the 
visualization  techniques  this  study  uses  to  summarize  organization  patenting  by  technology 
classes.  Fifth,  section  3.6  presents  the  visual  signature  of  each  of  the  four  patenting  patterns 
identified  in  this  study  and  defines  the  rules  this  study  uses  to  classify  patent  histograms  by 
pattern  type.  Chapters  4  and  5  addresses  the  possible  organizational  drivers  of  and  evaluative 
applications  for  these  patenting  patterns. 

3.2  Organizations  of  interest 

This  study  focuses  on  the  patent  portfolios  of  the  DOD  in-house  research  laboratories.  The 
DOD  research  laboratories  are  a  set  of  organizations,  run  by  the  military  services,  which  conduct 
and  fund  research  across  the  social,  physical,  engineering,  and  medical  sciences.  Depending  on 
how  one  draws  the  organizational  boundaries  and  the  type  of  work  one  includes,  there  can  be 
some  discrepancies  as  to  the  identities  of  the  DOD  research  “labs”.  For  example,  some  sources 
list  the  Army  Corps  of  Engineers’  seven  component  research  laboratories  separately  while  other 
sources  group  them  all  into  the  single  Engineer  Research  and  Development  Center  (ERDC). 

This  study  identifies  “laboratories”  based  on  the  level  at  which  patent  data  is  reported 
(described  below)  and  the  availability  of  budget  data  in  the  DOD  In-House  S&T  Activities 
Report  series.  The  DOD  laboratories  relevant  to  this  study  are  those  that  reported  in-house  S&T 
budgets  (DOD  budget  category  6.1  through  6.3)  between  1990  and  2010  (the  period  for  which 
patent  data  is  available)  and  which  have  received  at  least  six  patents  in  at  least  one  technology 
class  (the  reason  for  this  patenting  threshold  will  be  discussed  in  section  3. 6). 39 


39 

This  definition  excludes  organizations  that  only  issue  extramural  research  grants  such  as  the  Office  of  Naval 
Research  (ONR)  and  the  Defense  Advanced  Research  Projects  Agency  (DARPA),  contractor-operated  Federally 
Funded  Research  and  Development  Centers  (FFRDCs),  University  Affiliated  Research  Centers  (UARCs),  and 
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Based  on  the  DOD  In-House  S&T  Activities  Report  series,  there  are  36  separate  DOD 
research  “labs”  with  significant  amounts  of  in-house  S&T  work  during  the  study  period 
including  the  40 

•  Naval  Research  Lab  (NRL), 

•  Army  Research  Lab  (ARL), 

•  ten  Air  Force  Research  Lab  (AFRL)  directorates, 

•  five  Army  Research  Development  and  Engineering  Centers  (RDECs), 

•  Army  Medical  Research  and  Materiel  Command  (USAMRMC)  laboratories, 

•  Edgewood  Chemical  Biological  Center  (ECBC), 

•  Army  Corp  of  Engineering  Research  and  Development  Center  (ERDC)  (as  one  unit), 

•  Naval  Health  Research  Center  (NHRC), 

•  Naval  Medical  Research  Center  (NMRC),  and 

•  four  Naval  Warfare  Centers. 

However,  because  the  level  at  which  patent  data  are  reported  and  the  level  at  which  budget  data 
are  reported  do  not  always  agree,  some  “laboratories”  may  be  aggregated  or  disaggregated  for 
specific  analyses.  This  issue  primarily  affects  AFRL,  USAMRMC,  and  the  Naval  Warfare 
Centers.  While  the  terms  lab  or  laboratory  are  used  throughout  this  dissertation,  the  more 
appropriate  term  may  be  research  organization.  Patent  data  are  available  by  research 
organization,  and  while  all  labs  are  research  organizations,  not  all  research  organizations  are 
labs.41  Summaries  of  each  DOD  laboratory’s  technology  focus  areas  and  budgets  by  activity  type 
(e.g.  basic  research,  applied  research,  technology  development)  are  provided  in  Appendix  D. 


3.3  Data  sources 

The  primary  challenge  when  analyzing  DOD  laboratory  patents  is  matching  patents  to  the 
specific  originating  facility  as  opposed  to  the  owning  service.  Patents  derived  from  work 
performed  at  a  government  laboratory  (as  opposed  to  at  another  organization  which  received 
funding  from  the  laboratory)  will  usually  be  assigned  to  the  lab's  parent  agency  43  In  the  case  of 
patents  generated  in-house  at  DOD  labs,  the  assignee  is  the  service  (e.g.  United  States  as 


organizations  like  the  Marine  Warfighting  Lab  that  conduct  research  and  experimentation  into  doctrine  and  use  of 
technology. 

40 

For  comparison  note  that  the  Office  of  the  Assistant  Secretary  of  Defense  for  Research  &  Engineering 
(ASD(R&E))  listed  67  Service  Laboratories  in  the  on-line  Defense  Laboratory  Enterprise  Directory  as  of  November 
201 1  (ASD(R  &  E),  2012;  Defense  Science  Board,  2012).  The  In-House  S&T  Activities  Reports  aggregate  many  of 
these  67  laboratories  and  report  data  only  for  the  parent  organizations. 

41 

Credit  to  Susan  Marquis  for  pointing  out  this  distinction. 

42 

This  is  a  challenge  for  research  on  patents  from  any  U.S.  government  organization.  See,  for  example,  the  number 
of  assumptions  made  in  Jaffe  &  Lemer  (2001)  to  reduce  DOE  patents  to  DOE  FFRDC  patents. 

43 

The  assignee  is,  essentially,  the  entity  that  owns  the  patent  and  all  the  associated  rights.  If  the  lab  declines  to  file  a 
patent,  the  inventor  may  pursue  a  patent  on  his  or  her  own,  in  which  case  the  patent  will  be  assigned  to  the 
individual  inventor  or  a  company  of  the  inventor’s  choosing. 
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Represented  by  the  Secretary  of  the  Army).  In  theory,  the  originating  facility  could  be  inferred 
by  finding  the  DOD  laboratory  closest  to  the  inventors’  home  addresses,  which  are  listed  on  the 
front  page  of  a  patent.  However,  in  areas  where  there  are  several  DOD  facilities,  it  can  become 
difficult  to  identify  the  most  plausible  home  facility  for  an  inventor,  and  the  error  rate  of  this 
indirect  facility  identification  method  could  be  high.44  Therefore,  some  external  source  is  desired 
that  links  the  inventor  or  the  patent  itself  with  a  specific  research  facility.  This  study  relies  on  the 
list  of  patents  by  facility  at  the  DOD  TechMatch  website. 

3.3.1  DOD  TechMatch 

In  the  case  of  DOD  labs,  a  list  of  laboratory-related  patents  is  available  through  one  of  the 
partnership  intermediary  organizations  established  to  support  the  transition  of  DOD  technologies 
to  the  commercial  sector.  Partnership  intermediaries  are  state,  local,  or  non-profit  entities  that 
facilitate  cooperative  or  joint  ventures  between  federal  laboratories  and  small  businesses  or 
educational  institutions  (see  15  USC  §  3715).  Federal  laboratories  were  given  the  authority  to 
contract  with  partnership  intermediaries  in  the  Defense  Authorization  Bill  of  1991. 

DOD  TechMatch  is  a  partnership  intermediary  founded  in  2005  to  improve  the  data  available 
for  match-making  between  DOD  labs  and  commercial  users  of  their  technologies  (Morrison, 
2007).  TechMatch  maintains  an  on-line  information  center  for  DOD  patents,  licenses,  Small 
Business  Innovation  Research  (SBIR)  grants,  and  contracting  opportunities  (West  Virginia 
Technology  Consortium  Foundation,  2010).  In  order  to  direct  interested  private-sector  parties  to 
the  correct  contact,  TechMatch  employs  a  liaison  who  works  directly  with  the  DOD  laboratories 
to  link  patent  licensing  opportunities  to  specific  DOD  facilities  (West  Virginia  Technology 
Consortium  Foundation,  2012). 

DOD  TechMatch  lists  patents  by  number,  title,  and  originating  facility  for  58  DOD  "labs”.45 
In  addition  to  the  labs  identified  in  the  previous  section,  TechMatch  also  lists  patents  for 
organizations  that  are  not  S&T  performers  according  to  the  In-House  S&T  Activities  Reports. 
For  example,  TechMatch  lists  patents  for  the  Naval  Postgraduate  School  and  for  both  the  Naval 
Air  Systems  Command  as  well  as  its  subordinate  Warfare  Center  labs  at  China  Lake,  Point 


44 

If  the  inventor  of  an  Army-assigned  patent  lives  anywhere  in  North  Alabama  or  Southern  Tennessee,  it  is  likely 
that  the  inventor  works  for  the  Army’s  Aviation  and  Missile  Research  and  Development  Center  (AMRDEC)  at 
Redstone  Arsenal  in  Huntsville,  AL.  This  is  the  only  Army  facility  in  the  region  with  significant  in-house  research 
activities,  although  there  are  a  few  others  with  minor  research  expenditures.  However,  a  Navy  inventor  who  lives  in 
Northern  Virginia  could  plausibly  work  for  one  of  several  Navy  research  labs  including  the  Naval  Research  Lab 
(NRL),  the  Naval  Surface  Warfare  Center  (NSWC)  at  Carderock,  or  NSWC  Dahlgren. 

45 

As  discussed  above,  what  counts  as  a  lab  can  vary  by  source.  DOD  TechMatch  generally  lists  “labs”  at  a  level 
one  might  expect  from  organizational  charts,  e.g.  the  Office  of  Naval  Research  (ONR)  and  NRL  are  listed  separately 
while  the  Army  Research  Office  (ARO)  is  not  listed  as  a  separate  entity  from  the  ARL;  each  AFRL  directorate  is  a 
separate  lab,  but  all  of  the  USAMRMC  organizations  are  grouped  together  as  one  lab  (see  Appendix  D  for 
organizational  charts). 
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Mugu,  and  Patuxent  River.  DOD  TechMatch  also  includes  a  “TechMatch  Patent  Lab”  entry  for 
each  of  the  services  to  indicate  patents  for  which  they  could  not  identify  a  specific  laboratory. 

As  of  January  2012,  the  DOD  TechMatch  website  lists  the  patent  number,  title,  and  related 
laboratory  for  6,905  patents.  Note  that  TechMatch  lists  only  patents  issued  since  1990.  The  vast 
majority  of  these  patents  are  assigned  to  the  Army,  Navy,  or  Air  Force  and  most  likely  represent 
in-house  work.  Two-hundred  fifty-three,  or  about  4%  of  the  patents,  are  assigned  to  other 
entities.  A  plurality  of  these  otherwise-assigned  patents,  109,  are  assigned  to  individuals.  The 
remainder  are  assigned  to  various  other  entities,  including  large  U.S.  government  contractor 
organizations  like  General  Dynamics,  other  government  agencies  such  as  NASA  and  the 
Department  of  Energy  (DOE),  and  some  foreign  entities  such  as  Toyota  and  the  United 
Kingdom’s  Ministry  of  Defense.  These  patents  could  be  the  result  of  CRADAs  or  individual 
assignment  after  the  service  declined  to  patent.  All  of  these  patents  are  kept  in  the  dataset 
regardless  of  assignee  under  the  assumption  that,  given  TechMatch ’s  mission  to  connect  private- 
sector  organizations  with  in-house  developed  government  technologies,  the  patents  not  assigned 
to  a  military  service  were  still  created  in  close  collaboration  with  the  lab  (see  discussion  of 
CRADAs  below). 

The  TechMatch  dataset  also  includes  243  statutory  invention  registrations  (SIR).  These  are 
not  patents,  but  are  rather  published  patent  applications  for  which  the  applicant  has  waived  the 
right  to  a  patent.  SIRs  place  an  invention  in  the  public  domain  and  prevent  others  from  patenting 
similar  material.  SIRs  are  included  in  the  USPTO  database  with  an  'h'  at  the  beginning  of  their 
patent  number  (see  35  U.S.C.  157).  All  SIRs  are  also  retained  in  the  dataset  since  they  represent 
inventions  that  could  qualify  for  patents  if  not  for  some  prior  disclosure  (such  as  a  conference 
presentation)  or  the  desire  of  the  Service  to  place  the  technology  in  the  public  domain. 

3.3.2  Note  on  identifying  patents  resulting  from  extramural  or  cooperative  research46 

While  this  study  focuses  on  in-house  generated  patents,  the  majority  of  the  DOD 
laboratories’  budgets  go  to  extramural  research  (see  Appendix  D).  Patents  resulting  from 
extramural  defense  research  are  not  listed  in  TechMatch,  and  identifying  them  would  require 
additional  analyses  of  various  patent  and  research  contract  databases.  If  extramural  research 
results  in  a  patent,  following  the  Bayh-Dole  Act  of  1980,  the  patent  will  usually  be  assigned  to 
the  contractor,  university,  or  individual  who  performed  the  work.  In  such  cases,  it  may  be 
possible  to  link  the  assignee  to  their  funder  through  a  contract  number  listed  in  the  required 
statement  of  government  funding  at  the  beginning  of  the  patent  text.  Since  contractors  and 
grantees  are  required  to  disclose  to  the  funding  agency  any  patents  derived  through  government 
funded  research,  it  may  also  be  possible  to  ask  the  funding  agency  for  a  list  of  external  patents. 

46 

Background  information  on  patent  regulations  was  obtained  from  the  Federal  Laboratory  Consortium’s  (FLC) 
guide  to  Federal  Technology  Transfer  Legislation  and  Policy  (aka  the  Green  Book)  (Federal  Laboratory  Consortium, 
2009),  interviews  with  technology  transfer  and  patent  professionals,  and  summaries  of  legislation  in  Jaffe  &  Lemer 
(2001)  and  Hughes  et  al.  (201 1). 
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It  is  not  clear  how  successful  either  approach  would  be  in  practice.  While  several  databases 
exist  that  contain  both  patent  bibliographic  data  and  government  research  grant  information, 
attempts  to  link  patents  to  their  related  grants  generally  rely  on  matching  inventor  names  to  the 
grantee  names.  For  example,  Zucker  et  al  have  developed  an  extensive  database  of  U.S.  patents, 
highly-cited  articles,  and  NIH,  NSF,  Small  Business  Innovation  Research  (SBIR),  and  Small 
Business  Technology  Transfer  (STTR)  grants,  but  while  they  have  linked  organizations  and 
individuals  across  these  records,  they  do  not  appear  to  have  extracted  and  linked  grant  numbers 
from  relevant  patent  or  grant  texts  (Zucker,  Darby,  &  Fong,  201 1). 

Parents  generated  during  Cooperative  Research  and  Development  Agreements  (CRADAs) 
can  also  be  relevant  to  this  study  if  some  new  research  was  actually  performed  at  the  lab.  If  a 
patent  is  generated  during  a  CRADA,  then  the  assignee  will  depend  on  the  specifics  of  the 
CRADA.  Licensing  of  existing  intellectual  property  and  the  disposition  of  future  intellectual 
property  are  part  of  CRADA  negotiations.  There  have  been  cases  in  which  both  the  contractor 
and  the  laboratory  were  assignees  for  patents  generated  during  CRADA  work.  For  example, 
patent  5,989,194  for  a  “Method  and  apparatus  for  detecting  ocular  disease  and  abnormalities,” 
was  part  of  a  CRADA  between  the  U.S.  Army’s  Aviation  and  Missile  Research  Development 
and  Engineering  Center  (AMRDEC)  and  Vision  Partners,  L.P.  It  is  assigned  to  both  the 
commercial  partner  and  to  the  U.S.  Army.47  Identifying  the  specific  government  facility  involved 
in  the  CRADA  faces  the  same  challenges  as  identifying  the  originating  facility  of  in-house- 
research  generated  patents. 

3.3.3  Patent  class  and  bibliographic  data 

Although  the  TechMatch  dataset  identifies  DOD  laboratory  patents  by  number,  title,  and 
facility,  this  analysis  also  requires  more  detailed  information  for  each  patent,  particularly  a 
complete  list  of  technology  classes  and  the  application  year.  To  add  this  information,  the 
TechMatch  patent  list  is  merged  by  patent  number  with  the  USPTO's  2009  patent  grant 
bibliographic  data  file  and  the  Jan.  6,  2012  USPTO  master  classification  file.  These  were  the 
most  current  files  available  at  the  beginning  of  this  study. 


47 

This  CRADA  was  the  result  of  a  fortunate  coincidence  of  geography  and  hobby.  One  of  the  co-founders  of  Vision 
Partners,  L.P.  wanted  to  improve  the  early  diagnosis  of  lazy  eye  in  children.  Existing  detection  methods  took 
pictures  of  young  children’s  eyes  and  sent  them  to  an  expert  for  analysis.  The  co-founder  thought  there  might  be  a 
way  to  automate  the  screening  process  and  had  heard  that  AMRDEC,  which  was  only  a  few  hours  away,  may  have 
relevant  expertise.  At  a  Huntsville,  AL  Chamber  of  Commerce  meeting  (local  to  AMRDEC),  the  co-founder  struck 
up  a  conversation  with  an  Army  researcher  on  the  basis  of  a  shared  home  town.  The  researcher  was  from  Memphis, 
TN  where  Vision  Partners,  L.P.  was  based.  The  Army  researcher,  it  turned  out,  happened  to  be  a  professional 
photographer,  was  familiar  with  the  optics  of  the  eye  from  his  work  as  a  laser  safety  officer,  and  had  experience  in 
image  processing  from  automatic  target  recognition  research.  The  Army  researcher  thought  there  might  be  a  way  to 
exploit  the  phenomena  that  created  red  eye  in  photographs  to  automatically  evaluate  the  likelihood  that  a  subject  in 
the  photo  had  lazy  eye.  After  a  prior  art  search,  they  initiated  a  CRADA  that  was  ultimately  successful  in  producing 
a  prototype  and  a  patent  (Personal  Interview,  2012). 
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The  annual  USPTO  patent  grant  bibliographic  data  file  provides  the  patent  title,  inventors’ 
names  and  state,  assignee  name,  primary  technology  class,  application  date,  and  issue  date  for 
patents  issued  since  1976.  The  USPTO  master  classification  data  file  lists  the  associated  class, 
subclass,  and  patent  number  for  all  patents  ever  issued.  The  classification  file  is  updated 
quarterly  to  include  new  patents  and  changes  to  the  classification  system.  Merging  the  master 
classification  file  with  the  bibliographic  data  file  results  in  a  complete  list  of  patents  issued  since 
1976,  information  about  their  inventors  and  owners,  and  a  complete  list  of  primary  and  cross- 
reference  classes  (for  a  review  of  the  USPTO  technology  class  system  see  Appendix  C).49 

3.4  Data  summary  and  limitations 

The  2009  USPTO  bibliographic  data  file  lists  1 1,444  patents  issued  to  the  U.S.  military 
services  between  1990  and  2009.  After  merging  the  TechMatch  and  USPTO  datafiles,  6,725 
patents  remain  in  this  study’s  dataset.  Around  three  percent  of  the  original  6,905  TechMatch 
records  could  not  be  matched  to  a  USPTO  bibliographic  entry.  These  180  patents  are  all  dropped 
from  the  data  set.50  Of  the  remaining  patents,  in  92%  of  cases  (6,179  patents),  TechMatch  and 
the  USPTO  records  agree  on  the  associated  Service,  i.e.  the  patents  are  assigned  to  the  Service 
that  owns  the  TechMatch-identified  lab.  Assignees  associated  with  the  mismatched  cases  include 
both  different  services  and  private  entities.  Table  3.1  summarizes  these  statistics. 

Table  3.1  Summary  of  overlap  between  TechMatch  and  USPTO  records 


Category 

Patents 

Issued  to  U.S.  Military  Services  1990-2009 

1 1 ,440 

Listed  on  TechMatch  website  as  of  Jan  2012 

6,905 

Active  with  issue  date  1990-2009 

6,725 

Assigned  to  TechMatch  ID’ed  Service 

6,179 

Other  assignee 

546 

TechMatch  therefore  lists  a  little  more  than  half  (58%)  of  the  patents  issued  to  the  U.S. 
Military  Services  between  1990  and  2009.  The  exact  fraction  varies  by  grant  (or  application) 
year  and  Service.  Table  3.2  shows  the  percentage  of  patents  assigned  to  each  Service  that  are 
included  in  the  TechMatch  data  set  by  application  year  blocks.  Overall  TechMatch  includes  71% 
of  all  Army  patents,  68%  of  all  Air  Force  patents,  and  only  44%  of  all  Navy  patents  issued 


48 

The  bibliographic  data  file  also  comes  with  a  table  listing  all  citations  between  patents,  although  this  information 
was  not  used  in  this  study. 

49 

It  also  results  in  a  very  big  file.  The  fully  merged  text  file  is  over  1 .3  GB  in  size. 

50  One-hundred  seventy-seven  patents  in  the  TechMatch  database  were  issued  in  2010  or  later  and  therefore  did  not 
appear  in  the  2009  USPTO  bibliographic  data  file.  Three  patents  had  been  withdrawn. 
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between  1990  and  2005.  Navy  patent  coverage  improves  over  time,  while  Air  Force  patent 
coverage  declines  over  time.  TechMatch  contains  only  around  one-fifth  of  Navy  patents  with 
application  dates  between  1990  and  1997,  but  it  has  70%  of  Navy  patents  with  application  dates 
between  1998  and  2005.  TechMatch  includes  84%  of  Air  Force  patents  with  application  dates 
between  1990  and  1997  but  only  48%  of  Air  Force  patents  with  application  dates  between  1998 
and  2005.  Army  coverage  is  between  70  and  80%  in  both  time  periods  with  no  consistent  trend 
over  the  two  decades  of  interest. 

Table  3.2  Percent  of  service-assigned  patents  in  TechMatch  dataset  by  application  year  range 


Application  Year 

Army 

Air  Force 

Navy 

1990-1997 

70% 

84% 

22% 

1998-2005 

79% 

48% 

69% 

1990-2005 

74% 

70% 

44% 

Figure  3.1,  figure  3.2,  and  figure  3.3  compare  USPTO  and  TechMatch  coverage  for  the 
Navy,  Army,  and  Air  Force  respectively  by  application  year.  The  solid  red  area  in  the 
background  shows  the  number  of  patents  for  that  service  in  the  2009  USPTO  bibliographic  data 
file,  the  solid  green  area  in  the  foreground  shows  the  number  of  TechMatch  patents  for  that 
service,  the  blue  line  shows  the  percent  of  USPTO  patents  included  in  TechMatch  (right  axis). 

Figure  3.1  Total  reported  Navy  patents  USPTO  versus  TechMatch 
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Figure  3.2  Total  reported  Army  patents  USPTO  versus  TechMatch 
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Figure  3.3  Total  reported  Air  Force  patents  USPTO  versus  TechMatch 
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While  it  is  possible  that  the  remaining  Service  patents  were  filed  by  individuals  outside  the 
DOD  laboratory  system,  and  therefore  are  outside  the  mission  of  TechMatch,  it  is  also  possible 
that  most  of  the  missing  patents  were  filed  by  the  labs  and  TechMatch’s  data  collection  is 
incomplete.  None  of  the  current  employees  of  the  West  Virginia  Technology  Consortium 
Foundation,  the  consortium  that  ran  TechMatch  until  2011,  are  familiar  with  the  patent  data 
collection  effort,  making  it  difficult  to  determine  where  and  why  gaps  might  exist  in 
TechMatch’s  data  base  (West  Virginia  Technology  Consortium  Foundation,  2012). 

This  evidence  of  missing  data  suggests  that  the  TechMatch-based  data  set  is  not  a  census  of 
DOD  in-house  laboratory  patents  (issued  1990-2009),  but  is  instead  a  sample  of  DOD  laboratory 
patents  with  unknown  biases.  This  introduces  an  additional  source  of  uncertainty  in  interpreting 
the  laboratory  patenting  trends.  Where  appropriate,  statistical  tests  are  used  by  this  study  to 
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characterize  this  uncertainty.  Whether  due  to  low  propensities  to  patent  or  due  to  missing  data, 
some  targeted  laboratories  had  to  be  dropped  from  this  study  or  aggregated  with  a  higher 
organizational  unit  because  they  were  associated  with  too  few  patents  to  analyze  individually. 
Fortunately,  for  laboratories  with  sufficient  patents  in  the  TechMatch  data  set  there  is  some 
evidence  that  the  data  available  for  each  laboratory  is  at  least  consistent  with  their  full  range  of 
research  areas. 

3.4. 1  Summary  of  data  set  by  laboratory 

Figure  3.4  shows  the  number  of  patents  in  the  final  merged  dataset  by  lab  and  service  (see 
color  key)  after  dropping  unmatched  patents  and  cleaning  the  lab  names.  The  laboratories  are 
arranged  in  order  of  total  patents.  Note  that  the  AFRL  directorates  are  combined  together  in 
figure  3.4  because  several  of  the  directorates  are  associated  with  fewer  than  20  patents  each. 
Also  note  that  several  organizations  shown  in  figure  3.4,  particularly  the  Naval  Facilities 
Engineering  Service  Center  and  all  “labs”  with  fewer  than  20  patents,  are  not  included  in  this 
study.  They  are  excluded  either  because  they  are  not  in-house  S&T  performers  according  to  the 
DOD  In-House  S&T  Activities  Report  series  or  because  there  is  no  single  technology  class  in 
which  they  have  at  least  6  patents. 


30 


Figure  3.4  Number  of  patents  in  the  DOD  TechMatch  database  by  organization  and  service 
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Note:  Seven  patents  assigned  to  two  laboratories.  NUWC  =  Naval  Undersea  Warfare  Center;  NSWC  =  Naval  Surface 
Warfare  Center;  NAWC  =  Naval  Air  Warfare  Center;  SPAWAR  =  Space  and  Naval  Warfare  Systems  Command; 
ARDEC  =  Armaments  Research  Development  and  Engineering  Center;  CERDEC  =  Communications-Electronics 
Research  Development  and  Engineering  Center;  NSRDEC  =  Natick  Soldier  Research  Development  and  Engineering 

Center;  AMRDEC  =  Aviation  and  Missile  Research  Development  and  Engineering  Center;  TARDEC  =  Tank 

Automotive  Research  Development  and  Engineering  Center;  NVESD  =  Night  Vision  and  Electronic  Systems 
Directorate.  All  other  acronyms  defined  earlier  in  this  chapter. 

Despite  the  apparent  missing  data,  there  is  evidence  that  suggests  the  patents  included  in 
TechMatch  may  be  representative,  in  terms  of  classes  covered,  of  the  labs’  complete  patent 
portfolios.  A  simple  comparison  of  each  lab’s  most  frequent  patent  classes,  based  on  the 
TechMatch  patent  list,  and  that  lab’s  mission  shows  an  alignment  between  the  two.  For  example, 
the  Armaments  Research  Development  and  Engineering  Center  (ARDEC),  which  "[develops] 
advanced  weapons,  ammunition  and  fire  control  systems  for  the  U.S.  Army",  (“ARDEC  Home 
Page,”  n.d.)  patents  primarily  in  the  following  areas  (specific  patent  topics  in  parenthesis  for 
illustration). 

•  Ammunition  &  Weapons  (self-explanatory), 

•  Measuring  &  Testing  (weapon  sights  and  testing  weapon  quality), 

•  Aero/Astro  (for  ballistics), 
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•  Optical  Systems  (targeting), 

•  Communications  &  Electromagnetic  (EM)  devices  (fuses,  targeting,  and 
countermeasures),  and 

•  Buildings  (ammunition  cases  -  containers  are  a  subclass  within  the  buildings  subject 
group). 

In  contrast  the  Natick  Soldier  Research  Development  and  Engineering  Center  (NSRDEC), 
which  is  responsible  for  the  soldiers'  "survivability,  sustainability,  mobility,  combat  effectiveness 
and  quality  of  life,"  (“About  Natick  Soldier  RD&E  Center  (NSRDEC),”  n.d.)  patents  primarily 
in 

•  Resins  and  Rubbers  (polymers  and  solvents), 

•  Organic  Compounds  (also  polymers  and  solvents), 

•  Life  and  Agricultural  Sciences  (shelf-stable  foods  and  more  polymers), 

•  Aero/Astro  (parachutes),  and 

•  Apparel  and  Textiles  (self-explanatory). 

While  in  theory  the  labs  could  patent  only  spin-off  technologies  that  are  unrelated  to  their 
defense  missions,  prima  facie,  it  would  be  strange  if  the  technology  classes  covered  by  a  lab’s 
patents  and  the  lab’s  stated  mission  did  not  align.  That  this  dataset  includes  a  representative 
range  of  technology  classes  for  each  laboratory,  given  their  stated  missions,  increases  the 
confidence  that  any  missing  data  is  not  systematically  biased  by  technology  class.  The 
representativeness  of  the  technology  classes  covered  in  this  data  set  will  be  revisited  in  section 
3.7.2. 

3.4.2  Impact  of  a  restricted  issue-year  range  on  the  data  set 

In  addition  to  uncertainties  introduced  due  to  overall  missing  data,  there  is  uncertainty  with 
respect  to  trends  in  the  most  recent  patent  filings.  Because  of  the  different  date  ranges  covered  by 
the  various  source  files,  the  data  set  used  in  this  study  contains  only  patents  issued  between  1990 
and  2009.  Since  it  takes  an  average  of  3  years  for  a  patent  to  move  from  application  to 
issuance,51  this  dataset  should  contain  few  patents  with  application  dates  prior  to  1988  or  after 
2005. 

Figure  3.5  shows  the  number  of  patents  in  TechMatch  by  issue  year  (line)  and  application 
year  (area).  The  left  side  of  the  graph  shows  the  expected  pattern:  a  steep  drop  in  the  by- 
application-year  line  from  1990  to  1988  and  very  few  patents  with  application  years  of  1987  or 
earlier  (around  5%  of  the  total  data  set  have  application  years  prior  to  1988).  On  the  right  side  of 
the  graph,  the  overall  number  of  patents  in  this  dataset  actually  begins  to  decline  after  an  issue 
year  of  2006  and  an  application  year  of  2004.  Many  patent  applications  submitted  between  2005 


51  Traditional  Total  Pendency  as  of  August  2012  was  32.7  months  according  to  the  USPTO  statistics  dashboard, 
http://www.uspto.gov/dashboards/patents/main.dashxml,  accessed  Oct.  8,  2012. 
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and  the  present  were  therefore  likely  still  pending  at  the  time  of  TechMatch’s  last  data  collection 
and  therefore  do  not  appear  in  TechMatch’s  database  of  issued  DOD  patents. 

While  the  decline  in  available  patent  data  begins  earlier  than  expected,  this  truncation  has  no 
effect  on  the  results  other  than  to  make  them  less  current.  Figures  throughout  this  document  will 
shade  regions  from  2005  to  the  present  to  indicate  that  the  visible  trends  in  these  regions  are 
likely  incomplete. 

Figure  3.5  Total  TechMatch  patents  by  application  and  issue  year 


by  application  year  —  by  issue  year 
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3.5  Visualization:  creating  and  organizing  histograms 

Once  the  data  has  been  collected  and  cleaned,  the  next  step  in  this  methodology  is 
constructing  the  patent  histograms.  The  theories  and  measures  proposed  in  this  study  are  based 
on  analyzing  collections  of  histograms,  such  as  figure  3.6,  which  show  the  number  of  patents 
received  by  an  organization  per  technology  class  and  application  year.  The  bars  in  figure  3.6 
indicate  the  number  of  (ultimately  issued)  patents  filed  by  this  sample  organization  in  a  given 
year  in  the  sample  technology  class.  The  class  code  and  a  brief  description  of  the  technology 
class  are  given  in  the  bar  at  the  top  of  figure  3.6.  Note  the  shaded  area  on  the  right  side  of  the 
figure  indicating  that  the  trend  from  2005  through  the  present  is  likely  incomplete  as  explained  in 
subsection  3.4.2. 

Any  data  analysis  system  can  produce  these  types  of  histograms.  This  study  used,  and 
recommends  using,  the  ggplot2  graphical  package  (Wickham,  2009)  in  the  R  statistical 
computing  environment. 
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Figure  3.6  Sample  within-organization  histogram 
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3.5.1  Patent  Maps 

When  all  of  a  lab’s  patents  are  plotted  this  way  together  on  one  page,  such  visualizations 
provide  an  overview  of  a  laboratory’s  patenting  activities.  This  image  is  like  a  map  identifying 
the  lab’s  major  research  areas.  Therefore  this  study  calls  figures  such  as  figure  3.7  Patent  Maps. 
This  example  Patent  Map  covers  all  of  the  technology  classes  in  which  NRL  has  at  least  six 
patents.  Each  cell  in  figure  3.7  contains  the  same  features  described  above  for  figure  3.6.  Each 
cell  has  the  same  count  (vertical  axis)  and  time  (horizontal  axis)  scale,  making  comparisons 
between  the  timing  and  level  of  patenting  between  classes  easier. 

In  addition,  the  color  of  the  bars  in  each  cell  groups  the  technology  classes  into  broader 
subject  areas.  These  subject  area  groups  are  based  on  the  USPTO’s  “Classes  Within  the  U.S. 
Classification  System  (Arranged  by  Related  Subjects)”  document  which  groups  the  roughly  500 
technology  classes  into  approximately  30  subject  groups.52  The  subject  groups  are  ordered  and 
colored  such  that  similar  subjects  have  similar  colors  and  are  plotted  near  each  other. 

Note  that  not  all  subject  groups  will  show  up  in  the  Patent  Map  for  any  given  lab.  Being  a 
large  and  diverse  lab,  NRL  has  patents  in  many  subject  areas,  but  it  does  not  patent  significantly 
in  ships,  aeronautics  and  astronautics,  building  materials,  and  several  other  categories.  As  a 
result,  figure  3.7  includes  only  13  of  the  27  total  subject  groups. 

The  icons  in  the  top  left  comer  of  each  cell  mark  in  figure  3.7  the  type  of  pattern  each 
histogram  represents.  The  rest  of  this  chapter  explains  those  patterns. 


52  Available  at  the  USPTO  website  as  of  Nov.  2012 
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Figure  3.7  Patent  trend  diagrams  for  all  active  NRL  classes 
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3.6  Classify  histograms  by  pattern  type 

By  examining  the  patent  maps  for  the  DOD  laboratories,  this  study  has  identified  four  basic 
patterns  of  patenting  that  are  common  across  research  organizations.  Together  with  the  zero- 
patents  case,53  these  patterns  form  a  collectively  exhaustive  and  mutually  exclusive  set  of  within- 
organization  patent  trend  descriptors.  Each  cell  in  NRL’s  patent  map  (figure  3.7),  or  any  other 
patent  map,  can  be  identified  as  one  of  these  four  types  of  patterns.  Figure  3.8  shows  an  example 
of  each  pattern.54  A  short  description  of  the  technology  class  used  in  each  example  is  along  the 
right  side  of  the  diagram.  Icons  on  the  left  indicate  the  type  of  pattern. 

Figure  3.8  Patent  Trend  type  examples 
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53 

The  zero  patent  case  is  not  discussed  in  this  dissertation  for  two  reasons.  First,  given  that  there  are  around  500 
top-level  patent  technology  classes,  the  number  of  possible  zero-patent  cases  for  any  given  organization  is 
overwhelmingly  large.  See  section  3.7.1  for  more  on  the  rate  of  few  or  zero  patent  class  cases  in  the  DOD  patent 
portfolios.  Second,  the  question  of  why  an  organization  did  not  produce  patents  in  a  given  class  is  a  different 
question  than  why  it  produced  this  particular  pattern  given  that  it  produced  any  patents  in  this  technology  class.  This 
dissertation  focuses  on  the  latter  question. 

54 

Histograms  in  Figure  3.8  are  drawn  from  the  Naval  Research  Lab’s  portfolio. 
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This  study  selected  the  names  for  each  pattern  primarily  to  reflect  the  visual  shape  of  the 
patent  trend.  The  first  pattern,  called  steady-work,  shows  a  steady  and  generally  low  level  of 
patenting  that  remains  unchanged  over  the  years  in  the  dataset.  The  second  pattern,  called  a 
spike,  shows  an  isolated  burst  of  patenting.  In  contrast,  the  third  pattern,  called  a  swell,  shows  a 
gradual  increase  in  the  rate  of  patenting  over  time.  The  fourth  pattern,  called  a  submarine,  shows 
two  clusters  of  patents  separated  by  several  years  or  even  decades.  As  will  be  explained  in 
section  3.6.4,  the  fourth  trend  is  called  a  submarine  because  the  early  cluster  of  patents  were 
unpublished  and  hidden  from  view  (“underwater”)  for  many  years. 

The  remainder  of  this  section  describes  the  features  that  distinguish  each  pattern  and  the  rules 
this  study  uses  to  classify  patent  histograms  as  specific  patterns.  This  study  classifies  the  DOD 
lab  histograms  manually,  but  the  information  in  this  section  could  be  used  to  develop  an 
automated  classification  system.  Later  chapters  will  discuss  the  possible  research  activities  that 
generate  each  pattern  and  how  some  patterns  point  to  innovative  outputs. 

This  study  uses  three  steps  to  classify  a  patent  trend  histogram  as  one  of  the  four  patterns. 

The  first  step  in  all  cases  is  to  visually  inspect  the  shape  of  the  patent  trend  to  identify  the  most 
likely  categorization.  Second,  in  order  to  clearly  distinguish  spikes  from  steady-work  and  swell 
cases,  this  methodology  analyzes  the  distinct  networks  of  inventors  that  contribute  to  the 
sequence  of  patenting  under  consideration.  Spikes,  as  defined  in  this  study,  involve  a  single 
inventor  group  while  the  other  patterns  may  involve  multiple  groups.  Third,  this  method  employs 
an  additional  statistical  test,  the  CUSUM  test,  in  order  to  estimate  the  likelihood  that  an  apparent 
swell  trend  is  actually  not  due  to  random  variations  in  the  observable  data  alone.  The  following 
subsections  walk  through  the  descriptions  and  classification  rules  for  each  pattern  individually. 

3.6.1  Steady-Work 

The  steady-work  pattern  is  effectively  the  baseline;  the  majority  of  cases  (57%)  observed  in 
this  study  fall  into  this  category.  Steady-work  histograms  with  higher  total  numbers  of  patents 
appear  as  described  above:  a  consistent  one  or  two  patents  per  year,  every  few  years,  throughout 
the  data  set.  As  the  total  number  of  patents  filed  by  the  organization  in  the  class  decreases, 
steady- work  histograms  begin  to  look  more  random  with  each  individual  patent  filing  separated 
by  several  years  from  the  next.  Figure  3.9  shows  an  example  of  a  high-count  and  a  low-count 
steady- work  case  drawn  from  NRL’s  portfolio. 
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Figure  3.9  Steady-work  examples:  high-count  (left  -  14  patents),  low-count  (right  -  6  patents) 
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3.6.2  Spikes 

The  introduction  to  this  section  describes  a  patent  spike  as  an  isolated  burst  of  patenting.  The 
burst  of  patenting  is  isolated  in  the  sense  that  it  is  not  part  of  a  larger  increasing  trend  of 
patenting  such  as  the  swell  described  in  the  next  section.  Furthermore,  the  burst  of  patenting  is 
also  isolated  in  the  sense  that  it  is  primarily  attributable  to  a  single  inventor  group. 

An  inventor  group  is  defined  as  the  network  of  all  inventors  from  the  same  organization  that 
appear  together  on  a  patent  (i.e.  the  network  of  patent  co-inventors).  Each  distinct  network  is 
designated  as  a  separate  group.  One  might  expect  that  all  researchers  within  a  lab  would  be 
linked  together  by  at  least  one  person  in  a  "six  degrees  of  separation  in  the  research  lab"  effect. 
However,  while  there  are  sometimes  one  or  two  large  groups  of  inventors  within  a  lab  who  span 
a  wide  range  of  research  areas,  there  are  also  many  smaller  groups  of  three  to  five  inventors  who 
work  in  a  narrow  range  of  technologies  and  never  patent  with  people  outside  their  group.55 

Understanding  the  range  of  inventor  groups  involved  in  producing  a  given  patent  trend  is  an 
important  step  in  separating  spikes  from  steady- work  or  swells.  Whereas  in  steady- work  and 
swell  cases  the  organization  tends  to  have  multiple  inventor  groups  who  are  filing  patents  in 
similar  but  not  directly-related  areas,  in  spike  cases  the  organization  has  one  inventor  group  that 
has  filed  several  closely-related  patents  and  few  others  who  are  working  in  the  same  technology 
area. 

Figure  3.10  shows  the  differences  in  inventor- group  activity  between  a  spike  case  (left)  and  a 
swell  case  (right).  The  bars  are  colored  by  inventor  groups  (note  that  some  colors  are  used  to 


55  This  behavior  is  an  interesting  area  for  future  work,  and  may  already  be  more  fully  explored  in  the  literature  on 
scientific  collaboration  networks. 
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represent  several  groups).  In  the  spike  case,  all  of  the  patents  in  the  burst,  and  all  but  one  of  the 
patents  filed  by  this  organization  in  this  class,  are  filed  by  the  same  group  of  inventors.  In  the 
swell  case,  there  is  an  apparent  burst  of  patenting  in  1999,  but  it  is  made  up  of  patents  from 
several  different  inventor  groups.  Furthermore,  an  inspection  of  the  patent  titles  and  abstracts 
would  show  that  the  patents  in  the  spike  case  (except  the  first  from  the  other  group)  are  all 
related  to  the  same  invention  (a  particular  type  of  fiber  optic,  discussed  further  below)  while 
those  in  the  swell  case  cover  a  range  of  topics  within  the  Measuring  and  Testing  subject  area. 

Figure  3.10  Sample  spike  (left)  and  swell  (right)  patent  patterns  by  inventor  group 
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Since  the  visual  signature  of  a  patent  spike  -  a  burst  of  patenting  in  one  or  two  years  that 
sticks  out  above  the  rate  of  patenting  in  the  rest  of  the  class  -  can  occur  by  chance  in  a  noisy 
steady-work  case,  this  study  requires  at  least  three  related  patents  filed  by  the  same  inventor 
group  within  two  years  in  order  to  classify  the  visible  patent  trend  as  a  spike  rather  than  as 
another  pattern.  If  the  visible  spike  is  attributable  to  multiple  inventors,  and  the  rate  of  patenting 
in  the  class  otherwise  is  flat,  the  trend  is  classified  as  steady-work.56 

Distinguishing  spikes  from  other  patterns  therefore  requires  an  additional  data  preparation 
step  beyond  what  is  required  to  generate  the  basic  histograms:  labeling  patents  based  on  the 
group  of  inventors  to  which  they  belongs.  To  perform  this  step,  this  study  has  implemented  an 
automated,  name-matching,  network-tracing  algorithm  that  labels  each  patent  in  the  data  set  by 


56  Note  that  visible  spikes  that  are  part  of  an  overall  increasing  trend  are  classified  as  swells  regardless  of  the 
number  of  inventor  groups  active  in  the  apparent  spike  because  the  overall  increasing  trend  is  considered  dominant 
to  the  work  of  any  particular  inventor  group.  This  phenomena  is  discussed  further  section  3.6.3. 
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its  inventor  group  (a  number  between  1  and  X).57  In  addition,  because  the  inventor  names  have 
not  been  cleaned,  for  any  Patent  Trend  Diagram  in  which  there  may  be  a  spike,  the 
methodology  also  involves  manually  scanning  the  inventor  names  to  check  for  errors  in  group 
assignment. 

Spike  cases  must  not  only  come  from  the  same  inventor  group,  they  must  also  cover  closely 
related  subject  matter.  Confirming  that  the  patents  in  the  spike  case  are  indeed  closely  related 
follows  a  similar,  but  less  strictly  objective  process:  reading  the  titles  of  patents  in  a  potential 
spike,  which  on  several  occasions  have  been  nearly  identical,  or  reading  the  patent  abstracts  if 
the  titles  are  not  sufficiently  clear.59  While  inventor  groups  are  unambiguously  defined  -  specific 
inventors  either  do  or  do  not  patent  with  each  other  -  the  degree  of  similarity  in  title  or  abstract 
that  is  sufficient  for  a  spike  is  left  to  the  judgment  of  the  researcher. 

For  example,  all  of  the  patents  in  the  spike  example  in  figure  3.8,  except  the  first  in  1995, 
were  filed  by  Ishwar  D.  Aggarwal,  Jasbinder  S.  Sanghera,  and  one  or  two  co-inventors  drawn 
from  a  pool  of  10  recurring  individuals.60  As  shown  in  Table  3.3,  the  patents  in  the  spike  also 
have  similar  titles.  Many  of  the  titles  share  words  and  phrases  such  as  "producing"  or  "making" 
"core/clad  glass  optical  fiber".  This  degree  of  similarity  is  sufficient  to  call  this  set  of  1 1  patents 
a  spike. 


Application 

year 

1996 


Table  3.3  Patents  application  year,  number,  and  title  for  spike  example 

Patent  Patent  title 
number 

5735927  Method  for  producing  core/clad  glass  optical  fiber  preforms  using  hot  isostatic  pressing 

5779757  Process  for  removing  hydrogen  and  carbon  impurities  from  glasses  by  adding  a 

tellurium  halide 

5879426  Process  for  making  optical  fibers  from  core  and  cladding  glass  rods 
5900036  Multi-cylinder  apparatus  for  making  optical  fibers,  process  and  product 
6195483  Fiber  Bragg  gratings  in  chalcogenide  or  chalcohalide  based  infrared  optical  fibers 


57  Inventor  names  are  provided  in  the  USPTO  patent  grant  bibliographic  data  release,  although  they  are  in  a  separate 
file  from  the  main  bibliographic  information. 

58 

For  example,  an  inventor  may  include  his  or  her  middle  initial  on  one  patent  application  but  not  on  another. 

59  Patent  abstracts  are  available  at  the  USPTO  website  or  through  Google  Patents 

60  Interestingly,  the  effects  of  mentoring  may  be  visible  at  the  end  of  this  trend.  The  first  listed  inventor  of  the  final 
patent  in  this  group  is  one  of  Aggarwal  and  Sanghera's  early  co-inventors,  Brian  Cole.  Cole’s  patent  for  an 
application  of  chalcogenide  fibers  is  filed  with  a  co-inventor  who  had  not  previously  appeared  in  this  group. 
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Application 

year 

Patent 

number 

1997 

5846889 

5949935 

1999 

6021649 

2001 

6526782 

2003 

7197899 

2005 

7245424 

Patent  title 

Infrared  transparent  selenide  glasses 

Infrared  optical  fiber  coupler 

Apparatus  for  making  optical  fibers  from  core  and  cladding  glass  rods  with  two  coaxial 
molten  glass  flows 

Multi  heating  zone  apparatus  and  process  for  making  core/clad  glass  fibers 
Multi  heating  zone  apparatus  and  process  for  making  core/clad  glass  fibers 
Amplification  device  utilizing  thulium  doped  modified  silicate  optical  fiber 


Impact  of  missing  data  on  the  detection  of  spikes 

Note  that  missing  data  may  cause  some  actual  spikes  to  be  unobserved.  If  the  data  include 
two  closely-related  patents  from  an  inventor  group,  and  a  third  patent  exists  but  was  missing 
from  the  TechMatch  data,  then  the  laboratory  will  not  get  credit  for  that  spike  in  this  study.  This 
method  therefore  produces  a  lower  estimate  for  the  number  of  spikes  in  a  lab’s  portfolio. 

3.6.3  Swells 

Classifying  a  particular  patent  trend  as  a  swell  involves  all  three  classification  steps 
mentioned  in  the  introduction  to  section  3.6:  visually  identifying  a  likely  candidate,  examining 
the  inventor  groups,  and  testing  for  the  likelihood  that  the  observed  trend  is  not  due  to  missing 
data  or  noise.  First,  swells  involve  a  gradual  increase  in  patenting,  rather  than  concentrated  burst 
seen  in  swells.  Second,  in  swells  the  increase  in  patenting  is  usually  attributable  to  the  actions  of 
many  different  inventor  groups  rather  than  just  one.  Third,  the  increase  in  patenting  should  be 
sufficiently  large  as  to  be  statistically  different  from  the  result  of  random  fluctuations  in  the  year- 
by-year  rate  of  patenting.  The  following  subsections  expand  on  each  of  these  points. 

Visually  identifying  swells 

The  third  pattern  is  called  a  swell  because  of  the  sustained  rise  in  patenting  rates  shown  in  the 
histogram.  Also  note  that  the  cumulative  sum  line  for  these  cases  bears  a  resemblance  to  a  full  or 
partial  s-curve.  Figure  3.11  shows  a  swell  with  a  more  complete  s-curve-like  cumulative  sum. 

The  histogram  shows  the  number  of  ultimately  issued  patents  growing  at  an  increasing  rate  from 
1987  through  1999  and  then  decreasing  through  the  end  of  the  data  set.  The  cumulative  sum  line 
has  a  corresponding  s-curve-like  shape.  Note  that  any  observed  drops  in  the  rate  of  patenting  that 
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begin  around  2005  may  be  the  result  of  delays  in  the  processing  of  patent  applications  rather  than 
an  actual  decline  in  the  rate  of  patenting  at  the  organization. 

Figure  3.11  Swell  example  with  cumulative  sum  line 
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Examining  inventor  groups  in  swells 

Swells,  as  observed  in  this  study,  tend  to  be  the  result  of  multiple  inventor  groups. 
Occasionally  one  large  and  highly  productive  inventor  group  is  responsible  for  the  majority  of 
patents  in  a  swell,  but  in  these  cases  the  patents  tend  to  cover  a  wider  range  of  subjects  than 
would  be  the  case  in  a  spike.  Swells  can  also  include  small  spikes  within  the  larger  trend,  but  for 
classification  purposes  the  swell  trend  is  considered  to  be  dominant  pattern.  Swells  therefore 
have  the  connotation  of  a  sustained  program  in  contrast  to  the  spike's  more  project-like  profile. 

Examining  the  inventor  groups  in  swells  involves  the  same  process  as  examining  the  inventor 
groups  in  spikes.  Using  the  inventor  group  labels  described  above,  by-inventor-group 
visualizations  such  as  figure  3.10  will  quickly  show  whether  a  potential  swell  involves  few  or 
many  different  inventor  groups.  A  manual  scan  of  the  inventor  names  will  quickly  confirm 
whether  this  is  the  case.  If  one  inventor  group  appears  to  be  dominant,  then  the  person 
performing  the  classification  will  have  to  judge  whether  the  patents  are  sufficiently  related  to 
classify  the  pattern  as  a  spike  rather  than  a  swell. 

Using  change-point  detection  to  classify  swell  patterns 

Determining  whether  the  rate  of  patent  filings  is  trending  upwards  to  a  sufficient  extent  to  be 
classified  as  a  swell  as  opposed  to  noisy  steady- work  depends  primarily  on  the  researcher’s 
judgment.  The  limited  number  of  data  points  in  each  histogram  and  the  lack  of  a  known 
underlying  distribution  limits  the  types  of  statistical  methods  that  can  applied  to  this  situation. 
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However,  there  are  some  trend  detection  methods  that  can  assist  researchers  in  pattern 
identification.  This  study  uses  a  version  of  the  bootstrapped  CUSUM  change-point  detection 
method  (Taylor,  2000)  to  estimate  the  likelihood  that  the  mean  rate  of  patenting  shown  in  a 
patent  trend  histogram  is  changing  over  time.  Swell  patenting  patterns  should  show  an  increase 
in  the  mean  rate  of  patenting  over  time,  possibly  followed  by  a  decline  in  the  mean  rate. 

The  CUSUM  method  monitors  the  cumulative  sum  of  the  distance  between  each  observation 
and  the  sample  mean  in  a  sequence  of  observations  (Page,  1954).  The  first  step  is  to  calculate  the 
series  of  CUSUM  points  using  the  equation 

50  =  0;  Si  =  Si_1  +  Xt  —  X,  for  i  =  1,2, ... ,  #years 
where  X  is  the  mean  observed  annual  patenting  rates,  Xt  is  the  number  of  patents  received  by  the 
organization  in  year  i,  and  is  the  value  of  the  CUSUM  line  in  year  i.  If  the  observed  patenting 
rate  consistently  trends  below  the  mean  for  a  period,  the  CUSUM  line  will  trend  negative.  If  the 
observations  consistently  trend  above  the  mean,  the  CUSUM  line  will  trend  positive.  If  the 
observations  are  randomly  distributed  around  the  mean,  the  CUSUM  line  will  tend  to  be  flat. 

If  the  characteristics  of  the  underlying  distribution  are  known,  then  the  likelihood  of 
observing  a  trend  in  the  CUSUM  statistic  of  a  certain  size  under  the  null  hypothesis  of  a  constant 
generating  distribution  can  be  calculated  (Page,  1954).  If  the  underlying  distribution  is  not 
known,  as  is  the  case  here,  bootstrapping  can  be  used  to  estimate  the  likelihood  of  a  given 
CUSUM  trend.  This  study  uses  the  bootstrapping  method  described  in  Taylor  (2000). 

Taylor’s  approach  uses  as  the  test  statistic  the  difference,  Sd,  between  the  maximum  and 
minimum  value  of  the  CUSUM  line  5).  To  estimate  the  likelihood  of  observing  an  Sd  as  extreme 
as  calculated  for  the  observed  sequence,  the  method  randomly  reorders  (i.e.  bootstraps)  the 
observed  data  points  to  create  a  new  sample  drawn  from  the  same  underlying  distribution.  A 
CUSUM  line  and  Sd  are  calculated  for  each  new  sample.  This  procedure  is  repeated  multiple 
times,  Taylor  (2000)  recommends  1000  iterations,  to  produce  an  approximation  of  the 
cumulative  distribution  function  for  Sd.  The  likelihood  of  observing  a  test  statistic  smaller  than 
the  original  Sd  if  all  of  the  observed  data  points  are  randomly  drawn  from  the  same  distribution 
is  the  fraction  of  resampled  Sd  s  that  are  smaller  than  the  original,  i.e. 

n  number  of  resampled  Sd  <  Sd 

P(Sd  <  Sd)  = - 7~ - ,  .  . 

numer  of  resampling  iterations 

A  P(Sd  <  Sd)  above  a  certain  threshold  (e.g.  0.90,  0.95,  or  0.99)61  indicates  that  the 
observed  trend  in  the  CUMSUM  line  is  unlikely  to  occur  if  there  is  actually  no  change  in  the 
mean  rate  of  patenting  over  time.  Therefore,  the  null  hypothesis  of  a  constant  mean  should  be 
rejected.  This  study  investigates  all  histograms  with  a  P(Sd  <  Sd)  above  0.80  as  possible  swells. 
Note  that  the  CUSUM  is  calculated  using  only  patents  with  application  dates  between  1990  and 


61  Note  that  P(Sd  <  Sd  )  is  defined  here  and  in  Taylor  (2000)  in  the  opposite  direction  of  the  traditional  p-value.  P- 
values  usually  test  the  liklihood  that  of  a  test  statistic  greater  than  or  equal  to  some  threshold  instead  of  less  than. 
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2004  in  order  to  avoid  a  false  positive  caused  by  the  drop-off  in  data  at  the  edges  of  the  data 
sources’  date  windows. 

The  CUSUM  test  is  not  an  automatic  detector  of  swells.  Very  large  spikes  followed  by  low- 
count  steady-work  can  trigger  a  positive  result  as  can  low-count  steady-work  with  long  blocks  of 
zero  patents  followed  by  long  blocks  of  a  few  patents.  Figure  3.12  shows  the  patent  histogram, 
CUSUM  line,  and  P(Sd  <  Sd)  for  two  steady-work  cases  and  a  swell  case.  In  the  first  steady- 
work  case,  the  CUSUM  line  is  flat  and  P(Sd  <  Sd)  is  near  zero.  In  the  second  steady-work 
example,  the  few  patents  in  the  case  occur  in  clusters  that  trigger  the  detection  threshold.  In 
contrast,  the  third  example  shows  the  extent  of  trend  in  the  CUSUM  that  is  seen  in  a  large  swell. 

Figure  3.12  Sample  CUSUM  results  for  a  true  negative  (steady  work),  false  positive  (steady  work) 

and  true  positive  (swell)  case 


Steady  Work  -  true  negative 


Steady  Work  -  false  negative 


Swell  -  true  positive 


Note:  CUSUM  line  shown  in  red  over  the  patent  histogram  (black  bars).  CUSUM  probability  of  trend  shown  on  graph. 
Middle  case  is  classified  as  steady  work  because  maximum  rate  of  patenting  remains  below  2  patents  per  year. 


Because  of  false  positives  like  the  one  shown  in  the  middle  cell  of  figure  3.12,  the  CUSUM 
test  should  be  used  in  combination  with  a  visual  inspection  of  the  histogram  and  a  scan  of 
contributing  inventor  groups  and  patent  titles.  This  study  uses  the  following  two  rules  to 
determine  whether  a  sequence  of  patents  flagged  by  the  CUSUM  algorithm  is  a  swell  or  is  better 
classified  as  another  pattern. 

1 .  In  order  to  be  classified  as  a  swell,  the  number  of  patents  per  year  must  reach  three  or  more  in 
at  least  one  year.  Given  the  volume  of  possible  missing  data,  it  seems  unreasonable  to  call  a 
trend  of  one  or  two  patents  per  year  growth  even  if  this  low-rate  of  patenting  is  an  increase 
from  zero  in  the  early  years  of  the  data  set. 

2.  If  a  spike  (as  determined  from  a  scan  of  inventor  groups  and  patent  titles)  exists  in  the  class 
and  is  large  relative  to  the  number  of  non-spike  patents,  recalculate  the  CUSUM  without  the 
spike.  If  the  trend  remains  significant,  then  the  sequence  of  patents  is  likely  better  classified 
as  a  swell.  If  the  trend  disappears  when  the  spike  is  excluded,  then  the  sequence  is  better 
classified  as  a  spike. 
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Subject  to  these  two  conditions,  all  trends  with  a  P(5d  <  S%)  above  0.80  are  classified  as  swells. 
These  rules  will  classify  as  swells  patent  trends  that  increase  from  zero  to  three  patents  per  year 
near  the  end  of  the  date  range  and  patent  trends  that  include  of  one  or  more  spikes  within  an 
overall  increasing  rate  of  patenting. 

3.6.4  Submarines 

The  fourth  and  final  patent  trend  identified  by  this  study  is  the  submarine.  The  submarine 
trend  is  visible  in  the  Patent  Trend  Diagrams  because  of  the  difference  between  filing  (or 
application)  dates  and  issue  (or  grant)  dates.  The  filing  date  is  the  date  the  organization  submitted 
the  patent  to  the  patent  office.  The  issue  or  grant  date  is  the  date  the  patent  office  officially 
approves  the  patent.  This  dataset  only  contains  patents  that  were  issued  in  1990  or  later. 
Therefore,  excepting  special  circumstances,  it  should  include  no  patents  that  were  filed  earlier 
than  around  1980  because  they  should  have  been  already  issued  prior  to  1990. 62  The  submarine 
trend  highlights  when  one  of  those  special  circumstances  has  occurred.  In  the  DOD  context,  one 
likely  special  circumstance  is  the  patent  secrecy  order. 

In  the  submarine  example  in  figure  3.8,  there  is  a  cluster  of  patents  filed  in  the  1970s,  then  no 
patents,  and  then  another  cluster  of  patents  filed  in  the  mid-1990s  and  early  2000s.  However,  this 
does  not  mean  NRL  was  performing  no  work  in  this  technology  class  in  the  1980s.  The  lab  may 
have  other  patents  that  were  filed  in  the  1980s  that  were  also  issued  in  the  1980s  and  therefore  do 

/TO 

not  show  up  in  this  dataset.  The  early  cluster  of  patents  in  figure  3.8  were  filed  in  the  1970s  but 
were  not  issued  until  sometime  after  1990,  a  delay  of  20  to  30  years.  One  patent  in  figure  3.8  was 
filed  in  1958  but  not  issued  until  2003.  One  possible  cause  of  this  45  year  delay  is  secrecy.64 

At  the  request  of  a  defense  agency,  patent  applications  may  be  held  under  a  secrecy  order.  “A 
secrecy  order  withholds  the  grant  of  a  patent,  orders  that  the  invention  be  kept  in  secrecy  and 
restricts  filing  of  foreign  patent  applications,”  (Federation  of  American  Scientists,  1991).  When  a 
patent  application  is  under  a  secrecy  order,  the  USPTO  will  continue  to  process  the  application 
and  determine  whether  it  is  to  be  accepted  or  rejected,  but  they  will  not  publish  the  application  or 
actually  issue  the  patent  until  the  secrecy  order  is  lifted  (see  37  C.F.R.  §  5.3).  Secrecy  orders 
expire  after  one  year  unless  renewed  by  a  request  from  a  government  agency,  although  additional 


62  Recall  it  takes  on  average  about  3  years  and  sometimes  more  than  5  years  to  approve  and  issue  a  patent 

63  Also,  these  may  not  be  the  only  patents  the  lab  fded  in  the  1960s  and  1970s.  Deducing  the  explanation  for  this 
statement  is  left  as  an  exercise  to  the  reader. 

64 

In  the  private  sector,  it  was  possible,  until  2001,  to  create  a  similar  effect  by  constantly  revising  a  patent 
application.  Using  this  technique,  private  companies  could  maintain  a  trade  secret  but  also  establish  priority  in  the 
event  that  a  competitor  independently  discovered  and  attempted  to  patent  a  similar  technology.  In  2001,  the  patent 
office  began  publishing  all  applications  (except  secret  applications  per  above)  1 8  months  after  they  were  filed, 
removing  the  ability  to  keep  such  applications  secret.  Because  these  patents  with  long  delays  between  filing  and 
granting  would  stay  unpublished  and  hidden,  or  ’’underwater"  for  so  long,  these  types  of  patents  were  called 
"submarine  patents"  (Graham  &  Mowrey,  2004,  p.  447).  This  study  borrows  this  terminology  to  label  the  pattern 
that  shows  these  types  of  patents. 
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laws  may  still  prevent  the  disclosure  of  certain  information,  particularly  technical  data  that  has 
been  classified  by  the  DOD  (USPTO,  2012a). 

Note  that  patent  secrecy  orders  are  not  the  same  thing  as  Confidential/Secret/Top  Secret 
classification  by  the  U.S.  government,  although  a  patent  for  a  technology  that  has  been  classified 
as  Secret  will  likely  receive  a  patent  secrecy  order.  There  are  three  types  of  secrecy  orders.  The 
first,  and  least  restrictive,  identifies  patents  that  include  export  controlled  technologies.  Owners 
of  patents  covered  by  these  “Type  1”  secrecy  orders  may  disclose  the  content  of  their  patents  for 
“legitimate  business  purposes”  as  long  as  they  comply  with  applicable  export  control  laws 
(Federation  of  American  Scientists,  1991).  The  second,  Type  2,  secrecy  order  specifies 
procedures  for  handling  classified  material  at  the  USPTO.  Type  3  secrecy  orders  instruct  the 
USPTO  to  withhold  the  disclosure  of  patent  applications  that  are  not  officially  classified  but  do 
contain  information  that  could  be  detrimental  to  national  security  if  published  (see  Federation  of 
American  Scientists  (1991),  USPTO  (2012a)  chapter  0100  section  120,  and  25  USC  §  181). 

The  early  patents  in  the  submarine  example  are  likely  patents  that  the  government  filed  in  the 
50s,  60s,  and  70s,  but  were  covered  by  a  Type  2  or  3  secrecy  order  until  sometime  in  1990s.  All 
are  in  subject  areas  that  were  conceivably  classified.  The  earliest  patent  in  the  example,  filed  in 
1958,  is  for  a  concept  related  to  stealth  technology.  The  patents  in  the  main  group  are  related  to 
the  analysis  and  filtering  of  radar  side-lobes,  a  technique  that  was  related  to  several  radar 
improvements  being  developed  or  improved  in  the  1960s  and  1970s  including  over-the-horizon 
radars  and  phased  array  radars  (Chapman,  n.d.;  Howard,  n.d.). 

If  the  government  does  not  excessively  and  trivially  request  patent  secrecy  orders,  then 
submarine  patents  are  significant  inventions.65  However,  they  will  not  be  a  major  focus  of  this 
study,  primarily  because  submarines  are  a  secondary  effect.  Submarines  exist  within  technology- 
class  histograms  in  addition  to  the  steady-work,  spike,  or  swell  trend.  For  example,  the  more 
recently  filed  patents  in  the  submarine  case  in  figure  3.8  form  a  low-volume  steady- work  pattern. 
Therefore,  this  study  will  focus  on  explaining  and  using  the  three  primary  patterns:  steady- work, 
spikes,  and  swells. 

3.7  Constraints  and  Limitations 

As  mentioned  in  section  3.4,  the  data  set  used  in  this  study  does  not  contain  a  complete  list  of 
patents  for  all  DOD  labs.  This  missing  data  plus  natural  variations  in  patent  activities  in  a  lab 
each  year  introduces  some  uncertainty  to  the  pattern  classification  process  described  above.  The 
rule  that  spikes  include  at  least  three  related  patents  filed  within  two  years  by  the  same  inventor 
group  and  the  CUSUM  test  for  swell-related  shifts  in  the  mean  annual  patenting  rate  are 
designed  to  address  this  uncertainty.  However,  there  are  two  additional  limitations  that  should  be 


65  5321  secrecy  orders  were  in  effect  during  fiscal  year  2012  (Aftergood,  n.d.)  compared  with  253,155  patents 
granted  in  2012  (USPTO,  n.d.). 
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noted.  First,  this  analysis  is  performed  at  the  primary  class  level  rather  than  at  the  technology 
subclass  level.  This  has  the  effect  of  aggregating  a  laboratory’s  work  in  different  but  related 
technologies  into  one  trend.  Second,  by  focusing  only  on  active  classes  (those  in  which  a  lab  has 
at  least  6  patents),  this  study  implicitly  suggests  that  a  lab’s  patents  in  just  their  active  classes  are 
representative  of  their  primary  research  focus  areas. 

3.7.1  A  note  on  level  of  technology  aggregation 

Note  that  the  pattern  analysis  in  this  study  is  conducted  at  the  class  level  because  there  is 
generally  too  little  data  in  any  given  lab’s  subclass  portfolio  to  identify  any  pattern  other  than 
steady-work.  The  USPTO  defines  around  500  technology  classes,  with  around  150,000 
subclasses  defined  in  a  hierarchical  system  beneath  those  classes.66  Organizations  observed  in 
this  study  typically  have  fewer  than  5  patents  in  any  one  class,  and  only  in  the  largest 
laboratories  did  the  number  of  patents  in  any  one  class  reach  the  low-triple  digits.  (Figure  3.13 
shows  the  distribution  of  patents  per  class  in  each  laboratory’s  portfolio.)  Therefore,  since  there 
may  be  hundreds  of  subclasses  beneath  any  one  class,  most  organizations  will  have  few  to  no 
patents  in  any  given  subclass. 

Low  numbers  of  patents  per  subclass  mean  that  most  subclass  portfolios  will  appear  to  be 
low-count  steady  work  cases.  Even  at  the  class  level,  the  dominant  pattern  for  most  laboratories 
is  low-count  steady- work.  However,  in  theory,  all  of  the  patterns  described  here  from  the  class- 
level  data  may  also  exist  within  subclasses,  providing  a  more  detailed  view  of  an  organization’s 
work  by  showing  trends  within  more  narrowly  defined  technologies.  Since  the  subclass  structure 
is  hierarchical,  there  may  be  some  intermediate  subclass  level  that  still  provides  interesting 
insights.  Examining  the  relationship  between  the  hierarchical  structure  of  subclasses  and 
patenting  patterns  remains  an  area  for  future  work. 


66 


Based  on  an  analysis  of  classes  and  subclasses  listed  in  the  Jan.  6,  2012  USPTO  master  classification  file. 
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Figure  3.13  Distribution  of  patents  per  technology  class  for  DOD  TechMatch  dataset  (by  lab) 
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Note:  Horizontal  and  vertical  scales  vary  by  lab. 

Notice  that  in  Figure  3.13  there  is  a  break  in  the  distribution  around  5  patents  per  class  in 
many  cases,  and  in  all  cases  the  tail  of  the  distribution  is  reached  by  10  patents  per  class.  In  order 
to  focus  this  study  on  the  technology  areas  with  the  highest  volume  of  patenting,  this  study  only 
examines  technology  classes  in  which  the  lab  has  at  least  6  patents.  These  are  the  lab’s  “active” 
technology  areas.  The  term  “active”  is  used  to  distinguish  these  higher-patent-population  classes 
from  those  that  are  referenced  only  few  times  within  the  lab’s  patent  portfolio  and  therefore  may 
be  tangential  to  rather  than  a  core  part  of  the  lab’s  research  efforts.  While  there  can  still  be  a  lot 
of  noise  in  lab-class  combinations  with  fewer  than  10  patents,  indicated  by  frequent  jumps  in  the 
numbers  of  patents  per  year  or  long  gaps  between  the  appearance  of  patents  in  the  portfolio,  a 
threshold  of  6  patents  per  class  strikes  a  balance  between  a  more  complete  view  of  lab  activities 
and  having  sufficient  data  to  draw  inferences  by  class. 

Limiting  the  analysis  to  only  classes  in  which  a  lab  has  at  least  6  patents  can  delay  the 
detection  of  current  spikes  if  they  are  emerging  in  a  class  for  which  the  lab  has  little  or  no  prior 
work.  This  is  less  of  an  issue  if  the  spike  occurs  in  a  class  for  which  the  lab  already  has  an 
amount  of  steady  work  since  the  steady  work  patents  may  be  sufficient  to  register  the  class  as 
active.  However,  a  three-patent  spike  in  a  class  with  no  other  prior  work  will  go  undetected.  The 
detection  of  currently  rising  swells  would  be  delayed  even  without  this  restriction  since  the 
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pattern  is  unlikely  to  trigger  the  swell  detection  methods  described  above  until  the  total  volume 
of  patenting  in  the  class  exceeds  six  patents  total. 

3. 7.2  Validity  of  patent  trends  as  an  indicator  of  research  activity 

Patent  data  may  not  capture  all  or  even  most  of  the  research  activities  that  occur  at  a  DOD 
laboratory.  Section  3.4  notes  that  the  data  set  appears  to  include  a  range  of  patent  technology 
classes  for  each  lab  that  is  plausibly  representative  of  the  full  range  of  technology  focus  areas 
assigned  in  the  lab’s  mission.  This  establishes  some  confidence  that  a  patent-based  analysis  can 
encompass  a  representative  set  of  research  activities  for  the  labs.  A  further  check  on  the  validity 
of  interpreting  patent  trends  in  active  classes  as  indicators  of  the  primary  research  focus  areas  of 
the  laboratory  is  to  examine  the  fit  between  the  variety  of  active  classes  in  a  portfolio  and  the 
stated  mission  of  the  lab. 

The  DOD  laboratories  are  not  necessarily  free  to  pursue  any  technology  they  wish.  Many  of 
the  labs,  arguably  all  of  the  labs  except  ARL  and  NRL,  have  fairly  limited  missions.  If  the 
patent  trends  are  good  indicators  of  a  lab’s  primary  research  activities  -  as  opposed  to,  for 
example,  being  linked  to  spinoff  technologies  that  are  unrelated  to  the  lab’s  core  research 
mission  -  then  the  patent  portfolios  of  labs  with  broader  technology  missions  should  include 
more  active  technology  classes  than  those  of  labs  with  narrower  missions.  However,  this  analysis 
must  first  control  for  the  relationship  between  portfolio  size  and  the  number  of  active  classes. 

Figure  3.14  plots  the  laboratories  by  two  measures  of  portfolio  size:  total  patents  in  the 
dataset  (horizontal  axis)  and  the  total  active  classes  amongst  those  patents  (vertical  axis).  The 
circles  in  figure  3.14  mark  clusters  identified  with  the  k-nearest-neighbor  algorithm.68  The  line 
shows  the  best  linear  fit,  using  simple  linear  regression,  for  the  relationship  between  total  patents 
and  total  active  classes.  The  gray  area  shows  the  95%  confidence  interval  for  the  fit.  The  fit 
equation  and  R-squared  value  are  shown  as  well. 


67 

The  technology  focus  areas  of  each  of  the  DOD  labs  are  listed  in  Appendix  D. 

68 

The  k-nearest-neighbor  algorithm  minimizes  the  sum  of  square  distance  between  the  cluster’s  center  and  each  of 
its  members.  The  user  specifies  the  number  of  clusters  to  attempt.  See  documentation  for  the  R  package  cluster  or 
Hastie,  Tibshirani,  Friedman,  &  Franklin  (2005)  for  more  details. 
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Figure  3.14  Lab  clusters  by  portfolio  size  and  class  variety 
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Given  that  the  relationship  shown  between  total  patents  and  number  of  active  classes  is 
linear,  one  could  argue  that  the  main  driver  of  portfolio  class  variety  is  just  portfolio  size:  labs 
that  tend  to  patent  more  will  tend  to  have  patents  that  touch  on  more  technology  classes  and  will 
therefore  have  more  active  classes  in  their  portfolios.  However,  this  study’s  data  set  includes 
fewer  active  classes  for  any  given  lab  than  expected  given  the  number  of  patents  in  the  lab’s 
portfolio,  the  mean  number  of  technology  class  assignments  per  patent,  and  the  number  of 
technology  classes  that  exist.  Consider  the  following  example. 

While  any  single  patent  could  be  assigned  to  any  number  of  classes,  in  practice  the  average 
number  of  classes  per  patent  is  4.4  (Strumsky,  Lobo,  &  van  der  Leeuw,  2012,  p.  8).  If  every 
patent  in  a  lab’s  portfolio  had  the  average  number  of  class  references  and  each  reference  was 
different,  a  smaller  patent  portfolio,  such  as  the  one  observed  for  TARDEC  which  includes  only 
50  patents,  would  at  most  cover  220  classes.  A  larger  portfolio,  such  as  NRL’s  with  over  700 
patents,  could  in  theory  touch  on  the  entire  range  of  patent  technology  classes  (around  500).69 
Yet,  something  seems  to  have  a  strong  constraining  effect  on  the  actual  variety  of  classes 
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Note  that  this  argument  could,  in  theory,  go  in  either  direction.  While  including  more  patents  increases  the 
possible  number  of  classes  related  to  the  portfolio,  a  lab  that  wanted  to  have  a  portfolio  that  touched  on  more 
technology  classes  would,  by  the  same  logic  of  constrained  numbers  of  classes  per  patent,  need  more  patents  to  do 
it. 
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observed  in  a  portfolio  despite  the  total  number  of  patents.  TARDEC’s  automotive-focused 
patent  portfolio  actually  includes  only  1  active  (and  44  other)  classes.  NRL’s  portfolio  covers 
only  54  active  (and  101  other)  classes  rather  than  the  full  500. 

A  lab’s  technology  mission  may  be,  and  arguably  should  be,  a  strong  constraint  on  the 
variety  of  classes  in  the  lab’s  patent  portfolio.  Mission-based  constraints  can  plausibly  explain 
the  clustering  in  figure  3.14.  The  clusters  in  figure  3.14  can  be  described  in  terms  of  the  level  of 
class  variety.  NRL,  ARL,  and  NAVSEA  Newport  are  in  the  “high  variety”  group  with  54,  53, 
and  34  active  classes  respectively;  AFRL  and  several  Navy  Warfare  Centers  make  up  a  “mid 
variety”  group  with  20  to  30  active  classes;  and  the  Army  RDECs  and  the  rest  of  the  labs  fall  in 
the  “low  variety”  group  with  fewer  than  12  active  classes  each. 

The  clustering  of  labs  into  high  and  low  variety  groups  can  be  interpreted  as  a  clustering  of 
wide  and  narrow  mission  labs.  In  the  high- variety  group,  NRL  and  ARL  both  have  broad 
missions  that,  through  their  basic  research  portfolios,  can  touch  on  nearly  any  area  of  technology. 
The  Navy  Warfare  Centers,  which  fall  in  the  mid  and  high  variety  groups,  have  missions  that  are 
focused  on  types  of  Navy  systems  (air,  sea,  space  &  cyber),  but  within  those  domains  they  work 
on  a  range  of  technologies  including  computer  systems,  electrical  systems,  and  munitions.  In 
contrast,  the  Army  RDECs,  which  form  the  majority  of  the  low-variety  group,  could  plausibly  be 
described  as  having  responsibility  for  more  specific  technologies  in  a  narrower  set  of  classes 
such  as  electronics  and  sensors  (CERDEC),  food  and  clothing  (NSRDEC),  which  could  be 
considered  narrower  than  the  other  mission  descriptions. 

Given  these  clusters,  the  number  of  active  classes  in  each  DOD  laboratory’s  patent  portfolio 
seems  plausibly  representative  of  the  number  of  technologies  included  in  the  lab’s  mission.  This 
finding  increases  the  confidence  that  patents  in  the  lab’s  active  classes  can  be  interpreted  as 
representative  of  a  lab’s  primary  research  areas.  It  also  increases  the  confidence  that  this  data  set 
includes  a  representatively  complete  range  of  technologies  for  each  lab  -  with  the  possible 
exception  of  AFRL,  which  perhaps  should  be  in  the  high  variety  group  since  it  encompasses  all 
of  the  Air  Force’s  S&T  research. 

3.8  Final  Steps  -  from  patterns  to  innovations 

Visualizing  an  organization’s  patent  portfolio  as  technology  class  histograms,  and  classifying 
those  histograms  by  patenting  pattern,  can  clarify  the  type  of  work  a  research  organization 
performs,  the  organization’s  relative  emphasis  on  work  across  fields,  and  how  their  work  has 
changed  over  time.  This  is  useful  for  understanding  and  monitoring  the  focus,  or  technological 
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USAMRMC  and  ARDEC  are  outliers  in  this  group.  Both  USAMRMC  and  ARDEC  have  over  200  patents,  which 
suggests  they  should  be  mid- variety  labs  like  China  Lake,  but  their  total  number  of  active  classes,  1 1  and  10 
respectively,  are  closer  to  the  low- variety  labs.  With  approximately  10  fewer  active  classes,  perhaps  USAMRMC’ s 
medical  research  and  ARDEC ’s  armaments  research  are  related  to  fewer  patent  classes  than  China  Lake’s  aviation 
and  missile  research. 
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competencies,  of  a  research  organization.  However,  the  primary  goal  of  this  study  is  to  produce 
an  indicator  of  innovative  research. 

Technology-class  histograms  can  be  indicators  of  innovative  research  in  two  ways.  First, 
some  of  the  behaviors  that  may  produce  spikes  and  swells  suggest  that  the  organization  believes 
certain  research  products  will  become  widely  used,  one  of  the  criteria  for  an  innovation.  Second, 
independent  of  the  organization’s  hopes  for  their  work,  where  an  organization’s  patenting  pattern 
falls  relative  to  the  trend  in  the  field  as  a  whole  (i.e.  relative  to  the  pattern  made  by  all  patents 
filed  in  that  technology  class),  early  or  late,  can  be  an  indicator  of  innovation.  This  second 
method  builds  on  the  concept  of  technology  s-curves  in  the  patent  record  as  a  signal  of 
innovation  by  comparing  an  organization’s  entire  trend  to  the  field  s-curve  rather  than  comparing 
the  timing  of  individual  patents  as  done  in  Eusebi  (2011).  Chapter  4  develops  these  ideas  and 
explains  how  to  estimate  the  relative  locations  in  time  of  two  patenting  patterns. 

However,  in  addition  to  being  an  indicator  of  innovation,  an  organization’s  patenting  patterns 
could  also  be  used  as  a  tool  to  identify  and  monitor  the  organization’s  research  and 
commercialization  processes.  If  certain  research  approaches  are  more  likely  to  produce  spikes 
than  swells  or  steady  work,  then  a  lab  that  predominantly  employs  that  approach  should  have  a 
portfolio  with  a  preponderance  of  spikes.  By  this  reasoning  the  fraction  of  spikes,  swells,  steady 
work,  and  submarines  in  a  lab’s  patent  portfolio  can  be  used  as  an  indicator  of  its  particular 
approach  to  conducting  research.  If  some  approaches  are  more  desirable  to  managers  than  others 
-  for  example  using  a  near-term  versus  a  long-term  research  focus  -  and  those  approaches  have 
distinct  patenting  signatures,  then  the  pattern  fraction  can  be  used  as  a  monitoring  tool  to  assess 
the  fit  between  the  lab’s  actual  practices  and  the  desired  performance.  Chapter  5  demonstrates 
approaches  for  using  patenting  patterns  as  either  organizational  classification  tools  or  as  output 
measures  in  the  study  of  research  practices. 
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Chapter  4  Patenting  Patterns  as  Indicators  of  Innovation 


As  discussed  in  the  first  and  second  chapters  of  this  dissertation,  measuring  innovative 
research  performance  continues  to  be  a  challenge  in  the  study  and  implementation  of  science, 
technology,  and  innovation  policy.  Given  the  difficulty  of  observing  outcomes,  most  studies  and 
evaluations  of  government  research  organizations  rely  on  expert  evaluations  (aka  expert  or  peer 
review)  to  determine  the  health  and  performance  of  the  research  enterprise  (see  chapter  2).  To 
the  extent  that  patents  are  used  to  evaluate  government  research  organizations,  they  are  used  as 
bibliometric  indicators  of  knowledge  production  in  a  manner  similar  to  academic  journal  articles. 

This  study  proposes  an  alternate  approach  to  analyzing  the  patenting  trends  of  research 
organizations  which  more  directly  relates  to  an  organization’s  ability  to  produce  used 
innovations.  Chapter  three  introduced  four  patterns  that  can  be  observed  in  the  by-technology- 
class  patent  portfolios  of  research  organizations.  This  study  hypothesizes  that  these  patterns  are 
not  the  results  of  random  chance  but  are  instead  the  signatures  of  specific  research  and 
commercialization  activities  within  organizations.  More  specifically,  organizations,  this  study 
hypothesizes,  employ  the  activities  that  generate  patent  spikes  and  swells  when  they  are  pursuing 
technologies  they  believe  will  be  highly  useful.  If  innovations  are  defined  as  novel  and  widely- 
used  inventions,  then  under  this  hypothesis  patent  spikes  and  swells  are  indicators  of  potential 
innovations. 

Determining  whether  the  inventions  embodied  in  spike  or  swell  patents  were  actually  used, 
and  therefore  represent  actual  innovations,  requires  additional  information  beyond  observing  the 
existence  of  a  pattern.  Tracing  the  eventual  disposition  of  patented  inventions  through  interviews 
or  other  general  case  study  methods  would  perhaps  be  the  most  certain  way  of  confirming  use, 
but  such  methods  are  quite  resource  intensive  (Ruegg  &  Jordan,  2007).  A  faster  but  less  direct 
approach  is  to  locate  an  invention  along  the  technology  maturation  curve  (aka  s-curve). 
Inventions  that  emerge  near  the  take-off  inflection  point  of  their  technology  s-curves  are 
innovative  in  the  sense  that  they  were  early  and  possibly  foundational  contributions  to  a  field  that 
became  widely  used  (see  section  2.3  and  Eusebi  (201 1)).  However,  organizations  that  have  early 
patents  and  continue  to  increase  their  rate  of  patenting  during  the  technology’s  growth  stage  have 
likely  recognized  the  importance  of  the  technology  and  are  attempting  to  improve  their 
competitive  position  in  the  field  (see  comparison  of  German  and  Japanese  machine  tool  patents 
in  Ernst  (1997)).  Therefore,  the  relative  location  of  an  organization’s  patenting  pattern  along  the 
technology  class  patent  s-curve  can  be  an  indicator  of  the  actual,  rather  than  hoped-for, 
innovativeness  of  its  underlying  inventions. 

This  chapter  develops  these  ideas  in  two  phases.  Section  4.1  presents  a  theory  linking 
research  and  commercialization  activities  to  specific  patenting  patterns  and  uses  examples  from 
the  DOD  laboratories  to  illustrate  the  extent  to  which  spike  and  swell  research  activities  indicate 
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the  pursuit  of  innovation.  Section  4.2  demonstrates  how  to  compare  an  organization’s  patenting 
pattern  to  the  full-field’s  technology  s-curve  in  order  to  produce  a  measure  of  actual 
innovativeness. 

4.1  A  theory  of  patenting  behavior 

4.1.1  What  activities  generate  steady-work? 

As  an  introduction  to  thinking  about  the  activities  that  yield  different  patenting  patterns, 
consider  the  case  of  steady-work  patenting.  This  study  hypothesizes  that  the  steady-work 
patenting  pattern  is  the  result  of  most  organizations’  default  research  and  commercialization 
activities.  Within  the  DOD  laboratories,  the  majority  of  technology  class  histograms  (57%)  are 
steady-work  cases,  suggesting  that  steady-work  patenting  is  indeed  what  happens  most  of  the 
time.  The  theoretical  argument  for  this  hypothesis  is  as  follows. 

Research  organizations  have  certain  technology  areas  in  which  they  have  established  lines  of 
work  and  certain  established  rules  for  patenting  the  outputs  of  that  research.  If  neither  the  subject 
of  the  research,  the  intensity  of  the  research,  the  productivity  of  the  research  (including  sudden 
serendipitous  breakthroughs),  nor  the  intensity  of  patenting  changes,  then  there  should  be  no 
change  in  the  rate  of  patenting  from  year  to  year.  In  other  words,  if 

•  Dt0  is  the  amount  of  resources  (e.g.  dollars  and  people)  organization  o  puts  into 
researching  technology  area  t, 

•  Rt,0  is  the  rate  (per  year)  at  which  the  organization  can  turn  those  inputs  into  patentable 
discoveries, 

•  Pt;0  is  the  rate  (per  year)  at  which  the  organization  converts  research  discoveries  into 
patent  applications  in  class  t,  and 

•  At;0  is  the  rate  at  which  the  patent  office  accepts  and  grants  patents  to  organization  o  in 
class  t, 

then  the  number  of  patents  organization  o  receives  in  class  t  per  year,  Nt)0,  is  a  function  of  these 
terms. 

Nt,0  =  Rt,0 >  Rt,0>  ■A-t,o') 

A  change  in  any  one  term  will  change  the  rate  of  patenting  from  that  point  forward.  A  brief 
increase  in  the  rate  of  patenting,  which  then  returns  to  the  previous  rate,  produces  a  spike  pattern. 
A  consistent  compounding  increase  in  the  rate  of  patenting,  followed  by  a  leveling  off  at  the  final 
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higher  rate,  produces  the  swell  pattern. 
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Recall  that  this  study  includes  only  issued  patents,  therefore  the  patent  office’s  rate  of  acceptance  is  a  factor  in  the 
rate  of  patenting  visible  in  the  histograms. 
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Note  that  submarine  patterns  are  the  result  of  a  different  process.  As  explained  in  section  3.6.4,  in  submarines  the 
patent  office  delays  the  issuance  of  a  patent  for  many  years.  This  could  be  modeled  as  a  temporary  decrease  in  At>0, 
the  rate  at  which  the  patent  office  accepts  patents  from  the  organization.  However,  the  analogy  does  not  quite  fit 
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Spikes  and  swells  could  be  the  result  of  a  change  in  any  of  the  parameters  above.  For 
example,  if  an  organization  increases  the  dollars  and  people  employed  in  one  technology  area  for 
several  years  (increase  in  parameter  DU)),  the  amount  of  patentable  knowledge  generated  in  that 
technology  area  should  increase  each  year,  and  a  swell  would  appear.  Hiring  one  new  researcher 
who  is  interested  in  pursuing  a  particular  idea  could  temporarily  increase  an  organization’s 
research  productivity  (Rt,0)  in  a  given  technology  area  while  that  idea  is  played  out.  However, 
after  a  few  years,  that  researcher  may  return  to  some  lower  baseline  level  of  productivity  while 
searching  for  his  or  her  next  big  idea.  In  this  case,  a  spike  would  appear  in  the  organization’s 
patent  portfolio. 

The  rest  of  this  section  uses  brief  case  studies  of  DOD  research  projects  and  programs  to 
illustrate  more  specifically  how  changes  in  these  parameters  lead  to  spikes  and  swells.  The  case 
studies  use  data  collected  via  interviews  with  patent  attorneys  and  technology  transfer  personnel 
at  the  laboratories,  interviews  with  research  managers  or  the  inventors  themselves  when  possible, 
and  reviews  of  archive  material  including  the  patent  itself,  the  inventors’  CVs  and  online  bios, 
papers  and  books  written  by  the  inventors,  and  any  articles  that  talked  about  the  work,  for 
example  on  the  occasion  of  the  inventors  receiving  an  award. 

4.1.2  What  activities  generate  spikes? 

Recall  that  patent  spikes  are  clusters  of  three  or  more  patents  issued  within  two  years  to  the 
same  group  of  inventors  on  closely  related  material.  Single  inventions  may  be  protected  by 
several  patents  because  of  actions  initiated  by  the  patent  office  or  by  the  inventing  organization. 
Using  the  framework  presented  above,  changes  in  any  of  four  rates  can  yield  a  spike.  However, 
because  spikes  are  an  isolated  and  short-term  burst  in  patenting,  those  changes  are  only 
temporary  and  all  parameters  return  to  their  previous  levels. 

For  example,  if  the  laboratory  thinks  it  has  found  an  idea  with  large  commercial  potential, 
they  may  apply  more  resources  to  research  in  that  field,  experience  a  burst  of  productivity  as 
researchers  become  more  focused  and  inspired,  and  they  may  selectively  apply  for  patents 
related  to  that  technology  over  others.  However,  once  that  project  is  completed,  the  level  of  work 
in  that  technology  area  returns  to  its  former  level. 

Alternately,  decisions  made  by  the  USPTO  itself  may  create  a  spike.  The  patent  office  may 
decide  that  what  the  organization  sees  as  a  single  invention  is  actually  several  different 
patentable  inventions  according  to  patent  law  and  instruct  the  lab  to  file  separate  patents  for  each 
piece.  This  can  occur,  for  example,  in  biological  research  in  which  each  of  several  chemical 
compounds  is  a  separate  patent  even  though  they  are  being  used  together  for  a  common  effect. 
The  organization  may  be  trying  to  patent  the  use  of  the  chemicals  in  aggregate,  but  the  patent 


because  the  delay  applies  only  to  certain  patents  and  not  to  all  of  the  patents  the  organization  filed  in  that  class  that 
year. 
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Research  resources,  (Dt>0),  would  increase  slightly  as  well  in  this  scenario. 
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office  requires  a  separate  patent  for  each  compound  in  addition  to  the  combined  application 
(Personal  Interview,  20 12).74 

Regardless  of  whether  the  driving  factor  was  internal  or  external  to  the  lab,  patent  spikes 
represent  inventions  which  cost  the  lab  more  money  than  usual  to  protect.  Office  of  Research  and 
Technology  Applications  (ORTA)  personnel  interviewed  for  this  study  estimated  that  each 
patent  application  cost  the  lab  between  $5,000  and  $10, 000. 75  An  invention  covered  by  a  3- 
patent  spike  (the  smallest  possible  spike)  therefore  cost  the  lab  between  $15,000  and  $30,000  in 
intellectual  property  protection.  Even  if  multiple  patents  filings  were  requested  by  the  USPTO,  it 
is  reasonable  to  believe  that  the  lab  would  only  follow-through  with  a  $30,000  cluster  of  patents 
if  it  believed  that  it  would  get  a  higher  return  (financial  or  otherwise)  from  the  cluster  than  from 
a  single  $10,000  patent.  Based  on  this  reasoning,  this  study  hypothesizes  that  patent  spikes 
represent  inventions  that  the  lab  itself  believes  will  have  a  higher  military  or  commercial  impact 
than  the  average  invention. 

Consider  two  patent  spike  examples  identified  during  this  study,  one  from  NRL  and  one  from 
NAVSEA  Dahlgren.  In  both  cases,  research  related  to  military  requirements  led  to  discoveries 
with  sales  potential,  in  one  case  commercial  and  in  the  other  case  foreign  military.  The  ORTAs 
worked  with  the  researchers  to  create  a  cluster  of  patents  protecting  this  work  and  to  license 
these  patents  to  other  companies.  These  actions  can  be  described  in  terms  of  the  variables  used 
above:  researchers  increased  their  productivity  in  a  narrow  area  as  they  focused  on  particular 
breakthroughs  (Rt,0)  and  the  laboratory  patent  office  focused  their  applications  on  these 
particularly  promising  technologies  (Pt,0). 

Spike  Case  1:  NRL  Optical  Fibers 

This  case  was  already  briefly  described  in  the  discussion  about  inventor  groups  in  section 
3.6.2.  Aggarwal  and  Sanghera  were  pursuing  the  application  of  fiber  optics  in  several  Navy 
systems.  However,  to  meet  their  performance  needs  they  required  a  material  that  transmitted  at 
higher  wavelengths  than  traditional  silica  fibers.  Fibers  made  of  chalcogen  elements  (sulfur, 
selenium,  or  tellurium)  had  good  theoretical  properties,  but  their  manufacture  had  yet  to  be 
perfected  (Higby,  Aggarwal,  &  Friebele,  1994;  Sanghera,  Pureza,  Aggarwal,  &  Miklos,  1998). 
Aggarwal  and  Sanghera  developed  a  more  reliable  manufacturing  approach,  prototyped  it  at 
NRL,  and  then,  since  fiber  optic  manufacturing  was  not  a  core  NRL  mission,  licensed  the 
technology  to  industry  and  moved  on  to  other  projects  (Personal  Interview,  2012;  University  of 
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The  intellectual  property  literature  discusses  multiple  patent  strategies  (i.e.  patent  fences)  in  terms  of  their  ability 
to  increase  the  difficulty  a  competitor  faces  when  trying  to  invent  around  a  primary  patent  (see,  for  example, 
Gilardoni  (2007)  page  425).  The  story  told  here  about  the  patent  office  requesting  multiple  patents  to  cover  what  the 
lab  thought  of  as  one  invention  suggests  another,  less  strategic,  reason  why  firms  end  up  with  multiple  patents 
covering  closely  related  inventions. 
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The  ORTA  is  the  office  within  a  federal  research  organization  responsible  for  coordinating  technology  transfer 
activities  including  the  filing  and  licensing  of  patents.  For  authorizing  language  see  15  USC  §  3710  and  the 
Technology  Transfer  Act  of  1986  (Public  Law  99-502,  1986). 
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North  Carolina  at  Charlotte,  n.d.).  In  2012,  Aggarwal  and  Sanghera  received  the  Dr.  Arthur  E. 
Bisson  Prize  for  Naval  Technology  Achievement  which  recognizes  "individuals  who  have 
successfully  translated  research  findings  into  substantive  fleet  programs  that  meet  critical  Navy 
requirements,"  for  their  fiber  optic  and  transparent  ceramics  work  (NRL,  2012).  Their  work  in 
chalcogen  fiber-optics  forms  an  1 1  patent  spike  for  NRL  in  class  065  Glass  Manufacturing 
(Figure  3.10  left). 

The  Aggarwal  and  Sanghera  fiber  optic  work  is  an  example  of  a  coherent  project,  driven  by 
service  needs,  that  required  a  deep  understanding  of  fundamental  scientific  principles,  and 
which,  once  completed,  had  significant  commercial  or  military  impacts.  The  second  spike  case 
study  tells  a  similar  story  at  a  different  lab,  although  the  results  were  less  widely  used. 

Spike  Case  2:  NAVSEA  Dahlgren  Firearms 

Between  1988  and  1992,  two  engineers  at  the  Dahlgren  Naval  Surface  Warfare  Center, 
Michael  M.  Canaday  and  Fred  Watson  Jr.,  were  studying  several  design  problems  in  the  Mk  153 
Mod  0  Shoulder-Launched  Multipurpose  Assault  Weapon  (SMAW)  (Delguidice,  Monolo,  & 
Bechtel,  1997).  The  SMAW  was  originally  developed  by  McDonnell  Douglas  in  the  1980s,  with 
Dahlgren  assistance,  using  as  many  commercial  off-the-shelf  (COTS)  parts  as  possible 
(Delguidice  et  al.,  1997).  The  Mk  153  Mod  0  combines  a  British  spotting  rifle  with  an  Israeli 
rocket  launcher.  The  spotting  rifle  allows  the  user  to  test  the  aim  of  the  weapon  before  firing  the 
rocket.  The  SMAW  is  capable  of  firing  several  different  types  of  anti-structure  and  anti-armor 
warheads.  Problems  with  the  Mk  153  Mod  0  include  bubbled  launch  tubes,  sand  jamming  the 
triggers,  and  loss  of  alignment  between  the  spotting  rifle  and  the  rocket  launcher  (Delano,  2003). 
Rather  than  tweaking  the  Mk  153  Mod  0  design  to  address  each  issue  separately,  Canaday  called 
the  original  design  "kludged  together"  (Delano,  2003),  the  Dahlgren  engineers  decided  to  create 
an  entirely  new  design,  the  Mk  153  Mod  1. 

This  redesign  effort  resulted  in  one  prototype  and  15  patents  (Delguidice  et  al.,  1997).  All  15 
patents  do  not  appear  in  the  TechMatch  data  set,  but  four  of  these  patents  form  a  visible  spike  for 
Dahlgren  in  class  042  (Firearms)  in  1995.  Because  they  were  classified  in  both  class  042 
(firearms)  and  class  089  (ordnance),  some  of  the  Mk  153  Mod  1  patents  also  show  up  as  a  three 
patent  spike  in  Dahlgren's  class  089  trend  diagram  (Figure  4.1). 
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Figure  4.1  Dahlgren  SMAW  Patents  in  the  patent  trend  (colors  represent  inventor  groups) 
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Supporting  the  idea  that  inventions  that  are  covered  by  patent  spikes  are  more  commercially 
valuable  than  inventions  covered  by  only  a  single  patent,  a  case  study  of  this  invention  published 
by  Dahlgren  states  that  the  lab  pursued  a  "more  is  better"  patenting  approach  under  the 
philosophy  that  "a  strong,  complete  patent  portfolio  protects  a  product...  by  claiming  as  many 
new  features  as  possible,"  (Delguidice  et  al.,  1997).  This  strategy  is  sometimes  called  fencing  or 
surrounding.  In  patent  fencing,  a  firm  files  patents  for  the  core  invention  as  well  as  for  related 
lines  of  research  in  order  to  make  it  more  difficult  for  competitors  to  infringe  on  the  main  patent 
by  submitting  an  application  for  a  closely  related  invention  or  use  (Gilardoni,  2007).  However, 
Dahlgren  may  not  have  been  as  strategic  in  its  use  of  patent  fencing  as  some  commercial  firms 
since  the  Dahlgren  study  also  notes  that  "an  awful  lot  of  creativity  was  involved"  in  deciding 
which  specific  aspects  of  the  invention  should  be  protected  with  a  patent  (Delguidice  et  al., 
1997). 

While  the  U.S.  Marine  Corps  decided  not  to  pursue  the  Mk  153  Mod  1  (Delano,  2003),  CMS 
Inc.,  a  subsidiary  of  Deutsche  Aerospace,  approached  Dahlgren  about  licensing  the  Mod  1 
patents  and  forming  a  CRADA  (Delguidice  et  al.,  1997).  With  Marine  Corp  approval,  Dahlgren 
partnered  with  CMS  to  build  a  second  prototype  and  further  improve  the  Mod  1  design 
(Delguidice  et  al.,  1997).  Three  additional  patents  related  to  the  Mod  1  were  filed  in  1999, 
presumably  as  a  result  of  this  follow-on  work.  These  additional  patents  are  visible  as  second 
spikes  in  Dahlgren 's  class  042  (firearms)  and  class  089  (ordnance)  trend  diagrams  (see  Figure 
4.1).  CMS  eventually  sold  28  Mod  Is  to  Taiwan  (Delano,  2003). 
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4.1.3  Spikes  as  potential  innovations 

Both  of  the  examples  above  support  the  hypothesis  that  patent  spikes  are  the  output  of  highly 
successful  projects.  Both  of  these  cases  are  examples  of  projects  in  which  some  improvement  in 
performance  was  needed  to  meet  military  needs.  Not  only  did  the  laboratory’s  research 
succeeded  in  meeting  that  need,  but  the  results  were  also  sufficiently  promising  that  the  lab 
pursued  marketing  the  results  outside  the  service.  In  the  NRL  fiber  optic  case  the  need  was  more 
basic  -  a  new  material  with  certain  optical  properties.  In  the  Dahlgren  firearm  case  the  need  was 
more  applied  -  improvements  to  an  existing  weapon  system.  In  both  cases,  the  projects 
succeeded  in  their  aims,  and  the  resulting  knowledge  was  used,  although  use  was  more  extensive 
in  the  NRL  case  than  in  the  Dahlgren  case. 

It  is  not  a  given,  however,  that  all  spikes  indicate  technologies  that  are  actually  used.  The 
laboratory  expects  patent  spikes  to  have  a  higher  than  usual  chance  of  generating  revenue,  but 
there  are  several  factors  that  might  defeat  that  expectation.  For  example,  the  laboratory  could  be 
wrong  in  its  evaluation  of  the  market  opportunity;  a  technical  hurdle  could  be  discovered  during 
later-stage  development  or  manufacturing  that  reduces  the  profitability  of  the  invention;  or  the 
laboratory,  which  is  not  in  the  business  of  producing  and  selling  products  itself,  could  do  a  poor 
job  advertising  the  invention  to  potential  licensees. 

Therefore,  while  it  is  tempting  to  say  that  spikes  are  examples  of  innovations  because  the 
laboratory  expects  the  inventions  to  be  widely  used,  additional  work  would  be  necessary  to 
confirm  actual  use  of  these  inventions.  Still,  patent  spikes  represent  promising  technology 
transition  candidates.  Performing  case  studies  of  patent  spikes,  such  as  the  ones  described  above, 
could  be  a  more  focused  way  of  evaluating  a  laboratory’s  success  in  transition  discoveries  to 
users. 

4. 1.4  What  activities  generate  swells? 

This  study  hypothesizes  that  swells  are  the  result  of  long-term  increases  in  the  rate  of 
research  or  patent  production.  One  can  think  of  this  increase  as  being  directed  or  undirected. 
Directed  growth,  as  defined  here,  is  the  result  of  the  organization’s  leadership  determining  that  a 
particular  new  technology  will  be  important  and  purposefully  focusing  more  of  the  lab’s 
resources  on  that  technology.  Undirected  growth  occurs  when  researchers  independently  begin 
pursuing  a  new  technology  without  specific  and  explicit  leadership  guidance  and  without  directly 
coordinating  with  other  researchers.  For  example,  directed  growth  may  occur  when  Service  S&T 
leaders  decide  that  the  Service’s  research  should  focus  more  on  computer  networking-related 
technologies.  Undirected  growth  may  occur,  for  example,  as  researchers  search  for  ways  to  apply 
nanotechnology  to  their  research  under  the  anticipation  that  funding  agencies  will  increasing 
provide  resources  to  projects  that  involve  that  particular  concept. 

The  two  swell  cases  described  below  contain  elements  of  both  directed  and  undirected 
growth.  The  ARL  swell  in  solid-state  devices  has  a  more  directed  character  as  the  laboratory 
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used  its  director’s  discretionary  funding  stream  to  support  early  exploratory  research  in  a 
potentially  important  field.  The  NRL  swell  in  nanotechnology  seems  to  have  begun  in  a  more 
undirected  fashion  as  several  researchers  independently  decided  to  pursue  experiments  in 
nanotechnology.  However,  NRL’s  work  in  nanotechnology  may  be  becoming  more  tied  to 
customer-driven  work  with  the  establishment  of  the  Institute  for  Nanoscience  within  the  lab. 

Swell  Case  1 :  ARL  257  Solid-state  devices 

ARL's  work  in  solid-state  devices  class  257  (shown  in  figure  4.2)  is  diverse,  but  mostly 
related  to  the  creation  and  application  of  quantum  wells.76  While  it  is  unclear  if  technology 
foresight  at  the  laboratory  specifically  identified  quantum  wells  as  an  important  future 
technology,  several  of  ARL’s  solid-state  device  research  lines  began  as  Director’s  Research 
Initiatives  (“Army  Investment  in  Sensors  Research  Leads  to  Advances  in  Soldier  Protection,” 
2010)  or  other  internally- funded  exploratory  projects  (Barrick,  1990). 77  At  least  one  of  these 
lines  of  research  found  its  way  into  fielded  Army  systems.  Dr.  K.K.  Choi’s  work  on  quantum 
well-based  infrared  (IR)  detection  systems,  which  accounts  for  7  of  the  53  patents  in  this  trend, 
matured  over  20  years  to  provide  significant  cost  reductions  in  IR-detection  systems.  In  2010  Dr. 
Choi  received  the  Presidential  Rank  Award  for  dramatically  improving  the  performance  and 
reducing  the  cost  of  IR  systems  in  the  Army  (Spatchek,  2010).  Choi  filed  his  first  patent  for 
quantum  well-based  infrared  (IR)  detection  systems  in  1990. 


76 

Quantum  wells  are  subatomic  features  that  confine  electrons  such  that  they  can  only  respond  to  certain  energy 
levels  or  wavelengths  of  light.  Sandwiching  different  semiconductor  materials  together  can  produce  quantum  wells. 
These  features  can  be  used  to  produce  efficient  lasers  at  certain  wavelengths  or  sensors  that  are  sensitive  to  a  narrow 
range  of  wavelengths.  For  a  technical  summary  see  Levine  (1993). 

77 

Since  this  swell  starts  around  the  time  ARL  was  established,  much  of  the  foundational  work  in  this  swell  was 
probably  actually  done  in  the  Army  laboratories  that  were  consolidated  to  form  ARL  in  1991. 
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Figure  4.2  ARL  Swell  in  Solid-State  Devices 


12 

9 

I  6 

O 

3 

0 


o 

O 

o 

o 

o 

O 

CD 

Is- 

00 

G) 

o 

T — 

CT> 

a > 

a> 

G) 

o 

o 

CM 

CM 

Application  Year 


Swell  Case  2:  NRL  977  Nanotechnology 

NRL’s  small  swell  in  Nanotechnology  is  an  example  of  a  swell  generated  organically  by  the 
uncoordinated  and  not-necessarily-product-driven  work  of  many  researchers.  Figure  4.3  shows 
NRL’s  patent  trend  histogram  in  this  class.  While  the  total  number  of  patents  in  this  class  in  this 
dataset  is  small,  the  rate  of  patenting  at  NRL  in  class  977  has  clearly  been  accelerating  since 
1990,  placing  class  977  at  NRL  solidly  in  the  swell  category.  Feedback  from  research  managers 
at  NRL  suggests  that  the  growth  of  nanotechnology  research  and  patents  at  the  lab  happened 
organically  as  researchers  independently  pursued  various  academic  funding  streams.78  Based  on 
the  TechMatch  data,  nine  different,  non-overlapping  inventor  groups  are  active  in  class  977  at 
NRL.  Their  work  ranges  from  fundamental  research  into  the  production  of  nanotubes  and 
nanoparticles,  to  nano-scale  measurement  work,  to  medical  devices. 


78 

Interviewees  thought  the  work  represented  by  the  patents  tended  to  be  on  the  more  basic  side  of  NRL’s  portfolio 
and  was  therefore  sensitive  to  trends  in  science  funding. 
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Figure  4.3  NRL  swell  in  nanotechnology 
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While  this  swell  initially  seems  to  have  grown  as  the  result  of  independent  researcher 
activities,  nanotechnology  research  at  NRL  may  be  becoming  more  directed  toward  specific 
customer-driven  products,  and  this  swell  may  continue  to  grow  as  an  area  that  is  supported  by 
the  lab’s  own  internal  funds  and  technology  foresight.  In  2001,  NRL  established  the  Institute  for 
Nanoscience  in  order  to  better  support  interdisciplinary,  nano-related  research  at  the  lab.  The 
institute  receives  a  portion  of  its  funding  from  the  “core”  funds  NRL  receives  each  year  from  the 
Office  of  Naval  Research  (ONR).  This  money  helps  support  the  sophisticated  facilities  required 
to  investigate,  fabricate,  and  manipulate  nano-scale  materials.  These  facilities  are  open  to  any 
and  all  of  NRL’ s  research  staff,  providing  a  way  for  similarly- interested  researchers  to  connect 
with  each  other  as  well  as  providing  a  common,  already-paid- for,  high-tech  facility  to 
researchers  who  wish  to  apply  nanotechnology-related  techniques  to  their  projects  for  other 
customers. 

Saturated  Productivity 

The  rate  of  patenting  in  patent  swells  may  eventually  level  off  or  begin  declining.  An  analogy 
with  s-curves  may  be  appropriate  in  these  cases.  S-curves  have  two  inflection  points:  one  that 
marks  the  switch  to  increasing  growth  and  a  second  that  marks  the  switch  to  decreasing  growth 
(recall  the  discussion  in  section  2.3.2).  The  rate  of  patenting  in  a  swell  class  within  an 
organization  may  level  off  because  that  laboratory  has  reached  its  maximum  capacity.  For 
example,  to  continue  expanding  work  in  the  class  may  require  a  discontinuous  increase  in 
resources  such  as  constructing  a  new  building.  Alternately,  the  laboratory’s  managers  may  just 
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NRL  also  receives  transfers  through  the  Navy’s  Working  Capital  account  to  perform  research  for  Navy  and  other 
government  clients. 


977  Nanotechnology 


n - 1 - r 


r~ 


62 


feel  they  have  reached  a  sufficient  level  of  activity  given  their  goals.  Shifting  priorities  at  the 
laboratory  may  also  move  resources  from  a  mature  trend  to  a  new  area.  The  USPTO  may  also 
signal  that  work  in  the  area  is  saturated  by  decreasing  the  rate  at  which  they  accept  patents  in  that 
class  (again,  see  discussion  in  section  2.3.2).  Because  the  number  of  patents  in  this  dataset  begins 
declining  in  2005  (due  to  the  delay  between  application  and  issuance,  see  section  3.4.2),  it  can  be 
difficult  to  determine  if  many  of  the  swell  cases  have  actually  reached  saturation. 

4.1.5  Swells  as  potential  innovations 

Within-organization  swells  are  conceptually  similar  to  the  s-curves  observed  in  the  full  patent 
database  in  that  they  reflect  the  accumulation  of  increasing  amounts  of  inventive  effort.  In  the 
two  swell  cases  described  above,  the  within-organization  swells  were  driven  by  increasing 
numbers  of  researchers  pursuing  emerging  technologies:  quantum  wells  at  ARL  and 
nanotechnology  generally  at  NRL.  At  ARL,  at  least  some  of  this  research  resulted  in  improved 
military  devices.  At  NRL,  existing  expertise  in  nanotechnology  is  being  harnessed  to  support 
NRL’s  broader  research  portfolio  through  the  Institute  for  Nanoscience.  Note  that  within- 
organization  swells  could  also,  in  theory,  be  the  result  of  shifting  service  direction  which  shifts 
resources  from  an  old  to  a  new  technology  such  as  the  increasing  relative  emphasis  on  network 
technologies  in  the  Army  in  the  1990s. 

Continuing  the  analogy  between  swells  and  s-curves,  if  patents  in  the  early  part  of  the  field  s- 
curve  are  potential  innovations  because  they  represent  ideas  that  helped  trigger  the  rapid 
expansion  of  interest  in  the  field,  then  early  patents  in  the  within-organization  swell  are  also 
potential  innovations  because  they  are  the  foundations  on  which  the  rest  of  the  trend  are  built. 
Patents  that  both  start  a  within-organization  swell  and  lead  the  field  s-curve  are  doubly 
interesting  as  potential  innovations  because  they  are  potentially  used  both  within  the  lab  and  in 
the  wider  industry.  Section  4.2  continues  this  discussion  by  describing  how  to  visually  and 
mathematically  compare  the  timing  of  a  within-organization  swell  with  the  timing  of  a  field  s- 
curve. 

4.2  Identifying  Leadership  in  Patent  Trends 

Thus  far,  this  chapter  has  described  how  patent  trends  can  identify  what  a  lab  is  doing  but  has 
not  measured  the  value  of  those  activities.  One  way  to  approximate  the  value  or  impact  of  a  lab’s 
patented  research,  other  than  performing  multiple  case  studies,  would  be  to  look  at  the  relative 
number  of  citations  received  by  each  patent  in  a  trend.  However,  as  discussed  in  Appendix  B, 
several  uncertainties  remain  about  the  validity  of  forward  citations  as  a  measure  of  patent  impact. 
For  example,  citation  rates  vary  across  technology  areas  (B.  H.  Hall,  Jaffe,  &  Trajtenberg,  2001), 
interactions  occur  between  the  length  and  breadth  of  a  patent’s  claims  and  the  number  of 
citations  it  receives  (Eusebi,  2012),  and  the  actual  correlation  between  citation  rates  and 
commercial  impact  is  still  under  study.  An  alternative  approach  could  be  to  compare  the  timing 
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of  a  single  patent  application,  or  the  entire  trend,  with  the  evolution  of  patent  filings  in  the  field 
as  a  whole  (i.e.  against  all  patents  in  that  class). 

Several  authors  have  suggested  using  the  evolution  of  s-curves  in  the  patent  record  as  a  tool 
for  identifying  emerging  and  potentially  disruptive  technologies  (Eusebi,  2011)  and  as  a  tool  for 
aligning  technology  development  strategies  to  emerging  market  conditions  (Andersen,  1999; 
Ernst,  1997).  These  suggestions  are  based  on  the  theory  that  the  rate  of  patent  filings  within  a 
technology  class  does  not  accelerate  unless  the  earlier  ideas  in  the  class  are  used  and  diffused 
(Andersen,  1999,  p.  5).  When  the  cumulative  number  of  patent  applications  in  a  technology  class 
enters  the  growth  phase  (i.e.  begins  to  “take  off’),  it  reflects  a  sudden  increase  in  the  number  of 
parties  betting  that  the  technology  will  be  valuable  (Eusebi  &  Silberglitt,  n.d.).  Presumably  this 
expectation  is  rational  and,  for  at  least  some  of  the  patentees,  is  realized. 

Under  this  theory,  individual  early  patents  may  be  more  innovative  than  later  patents  in  the 
sense  that  their  inventors  were  early  movers  into  an  area  that  became  popular,  widely  used,  and 
profitable.  Applying  this  theory  to  the  patent  patterns  suggests  that  patent  spikes  that  lead  the 
field  s-curve  (occur  relatively  near  the  field’s  take-off  inflection  point  -  a  more  specific 
definition  leadership  is  provided  below)  are  cases  in  which  an  organization  developed  a  prescient 
and  potentially  innovative  invention  that  contributed  to  the  growth  of  the  technology  class. 

Yet,  if  an  organization  had  an  early  breakthrough  (represented  by  an  early  spike  or  steady- 
work  trend)  in  a  field  that  later  strongly  took-off,  why  did  the  organization  not  accelerate  their 
research  investment  along  with  everyone  else?  Did  the  lab  underestimate  the  value  of  their 
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discovery?  Did  the  lab’s  mission  constrain  their  follow-on  work? 

In  contrast,  when  an  organization  has  a  patent  swell  that  leads  the  field  s-curve,  this  suggests 
that  the  organization  was  not  only  an  early  mover  but  also  recognized  the  potential  of  the  field 
early  and  continued  to  expand  their  investment  in  the  field.  Because  of  this  extra  connotation  of 
innovativeness,  this  section  focuses  on  methods  that  identify  leading  and  lagging  swells  in  an 
organization’s  portfolio  as  an  indicator  of  innovative  behavior. 

4.2. 1  Visualizing  swell  leadership 

Identifying  when  an  organization’s  patent  swell  leads  the  development  of  an  s-curve  in  the 
field  as  a  whole  requires  some  method  to  compare  the  relative  timing  of  patents  in  these  two 
different  distributions.  In  his  ongoing  research,  Eusebi  uses  a  disease-outbreak  CUSUM-based 
model  to  estimate  when  the  s-curve  in  a  technology  subclass  begins  to  take-off  and  compares  this 
date  to  the  filing  date  of  an  organization’s  patents  in  that  subclass.  The  more  patents  an 
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Many  reviewers  have  suggested  that  spikes  could  be  characterized  as  “swells  that  didn’t  take”,  but  there  may  be 
alternate  explanations  for  the  appearance  of  spikes  versus  swells  in  a  given  part  of  an  organization’s  portfolio  that 
have  little  relationship  to  pursuing  the  field  trend.  However,  regardless  of  whether  the  lab  “should  have”  pursued  the 
s-curve  trend  in  that  class,  patent  spikes,  as  defined  in  section  4.1.2  are  still  inventions  that  the  laboratory  believes 
have  the  potential  to  make  a  commercial  impact.  While  the  lab  may  or  may  not  be  correct,  patent  spikes  could  be  a 
good  place  to  look  for  performance  “nuggets”  -  stories  of  specific  high-impact  inventions  used  to  illustrate  a  lab’s 
contributions  during  performance  reviews. 
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organization  has  before  the  take-off  date,  the  bigger  a  leader  and  innovator  it  is  considered  to 

O  1 

be.  This  method  works  to  compare  the  relative  innovativeness  of  two  (or  more)  organizations, 
but  this  study  attempts  to  develop  a  measure  that  can  determine  leadership  and  innovativeness  in 
a  more  absolute  sense.  Therefore,  this  study  compares  the  relative  timing  of  an  organization’s 
entire  patent  trend,  not  just  the  leading  edge,  to  the  entire  trend  in  the  technology  class. 

A  first  diagnosis  can  be  done  visually,  although  this  comparison  is  difficult  to  do  with  the 
patent  trend  histograms  because  any  one  organization  has  many  fewer  patents  in  a  single  class 
than  have  been  filed  in  the  field  as  a  whole.  Fortunately,  a  slight  transformation  of  the  data  can 
enable  a  direct  comparison  of  the  timing  of  patent  filings. 

To  illustrate  how  a  transformation  of  data  can  make  easier  the  comparison  of  patent  trends 
between  differently  sized  organizations  over  time,  consider  the  case  of  IBM’s  patenting  in 
nanotechnology.  IBM  has  the  reputation  of  being  a  pioneer  in  nanotechnology.  IBM  researchers 
were  responsible  for  some  of  the  enabling  discoveries  and  inventions  in  nanotechnology 
including  the  Scanning  Tunneling  Microscope  (at  IBM  Zurich  in  1981)  (Binnig  &  Rohrer,  2000), 
the  first  precise  positioning  of  individual  atoms  (to  spell  out  IBM  at  IBM  Almaden  in  1989) 
(Shankland,  2009),  and  the  independent  co-discovery  of  single-walled  carbon  nanotubes  (at  IBM 
Almaden  in  1993)  (D.  Bethune  et  al.,  1993;  D.  S.  Bethune,  2002).  These  early  discoveries  were 
patented  (e.g.  U.S.  Patent  4,343,993  for  “Scanning  Tunneling  Microscope”  and  U.S.  Patent 
5,424,054  for  "Carbon  Fibers  and  Method  for  their  Production”).  Today,  IBM  is  the  largest 
holder  of  U.S.  nanotechnology  patents.  Yet  its  patent  holdings  are  dwarfed  by  the  global 
expansion  of  nanotechnology  patenting  over  the  last  20  years. 

The  left  side  of  figure  4.4  shows  the  histogram  of  IBM's  patents  in  nanotechnology  and  the 
right  side  of  figure  4.4  shows  the  histogram  for  the  entire  field  in  nanotechnology  (class  977). 
Note  the  very  different  vertical  scales  in  each  figure.  Up  to  1200  patents  have  been  submitted  per 
year  to  the  USPTO  in  nanotechnology  (right  half  of  figure  4.4)  while  the  maximum  filed  per  year 
by  IBM  in  the  same  class  (left  side)  is  only  20.  If  both  curves  were  shown  on  the  same  graph 
with  the  same  scale,  IBM’s  curve  would  barely  be  visible. 

Figure  4.4  IBM  (left)  and  all  USPTO  (right)  patents  in  class  977  Nanotechnology 


IBM  Nanotechnology  Histogram  USPTO  Nanotechnology  Histogram 
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Eusebi  and  Silberglitf  s  work  comparing  the  leadership  rates  of  various  organizations  has  not  yet  been  published. 
That  work  is  currently  the  subject  of  a  RAND  project.  This  author’s  understanding  of  that  work  is  derived  from 
several  interim  briefings  by  Eusebi  and  Silberglitt  given  under  the  auspices  of  that  project. 
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IBM  Nanotechnology  Histogram 


USPTO  Nanotechnology  Histogram 
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Transforming  the  histograms  into  density  plots  can  enable  a  direct  comparisons  between  very 
differently  sized  patent  portfolios.  A  density  plot  normalizes  the  number  of  patents  filed  per  year 
in  a  histogram  such  that  the  total  area  under  the  curve  equals  one.  The  gray  background  area  in 
figure  4.5  shows  the  fraction  of  total  U.S.  nanotechnology  patents  filed  in  a  given  year 
(excluding  IBM’s  patents).  The  green  transparent  foreground  area  in  figure  4.5  shows  the 
fraction  of  IBM's  total  nanotechnology  patents  filed  each  year.  These  areas  are  equivalent  to 
areas  from  figure  4.4,  only  scaled. 


Figure  4.5  Density  plot  comparing  IBM  and  all  USPTO  patent  rates  in  nanotechnology  (class  977) 
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The  relative  location  of  IBM’s  (green  transparent  foreground)  curve  in  Figure  4.5  suggests 
that  their  patenting  activities  led  the  rest  of  the  field  by  several  years,  as  one  would  expect  given 
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their  role  in  early  nanotechnology  research  and  invention.  While  the  patent  density  curves  can  be 
used  to  compare  a  lab’s  trend  to  the  field  s-curve  in  this  manner,  relying  on  the  visualization 
alone  is  both  imprecise  and  time  consuming.  It  is  imprecise  because  of  both  the  limitations  of 
visual  acuity  and  the  potential  errors  introduced  by  missing  data.  A  faster  and  more  precise 
classification  approach  is  to  use  the  Mann- Whitney- Wilcoxon  (MWW)  Rank  Sum  Test  to 
determine  the  likelihood  that  the  organization’s  density  curve  lies  above  and  to  the  left  of  the 
field  density  curve. 

4.2.2  MWW  statistical  test  for  leadership 

The  Mann- Whitney- Wilcoxon  (MWW)  Rank  Sum  Test  first  combines  all  observations  from 
the  groups  of  interest,  places  each  observation  in  order  along  some  parameter  of  interest,  and 
assigns  each  observation  a  number  from  1  to  N,  called  its  rank  order,  where  N  is  the  total  number 
of  observations.  In  this  case,  the  patents  issued  to  all  organizations  are  ordered  by  application 
year.  For  example,  suppose  there  are  two  organizations,  each  with  four  patents  as  shown  in  Table 
4.1.  Placing  the  patents  in  order  of  application  year  results  in  the  rank  numbers  in  the  right-most 
column.  Notice  that  patents  with  the  same  application  year  are  assigned  their  average  rank. 


Table  4.1  Example  data  for  the  Mann-Whitney-Wilcoxon  Rank  Sum  Test 


Patent  Number 

Application  Year 

Organization 

Rank 

X00001 

1990 

A 

1 

X00002 

1991 

A 

2 

X00003 

1992 

B 

3 

X00004 

1993 

B 

4.5 

X00005 

1993 

A 

4.5 

X00006 

1994 

B 

6 

X00007 

1995 

A 

7 

X00008 

1997 

B 

8 

Next,  the  test  adds  together  the  total  rank  for  each  organization.  In  this  example,  the  “rank 
sum”  for  A  is  Wa  =  1+2+4. 5+7  =  14.5  and  the  rank  sum  for  B  is  Wb=3+4. 5+6+8  =  21.5.  The 
rank  sum  of  one  group,  say  group  A,  is  the  test  statistic.  Note  that  the  rank  sum  for  the 
comparison  group,  WB,  is  fully  determined  by  the  number  of  observations  N  and  the  rank  sum 
for  the  group  of  interest,  Wa.  The  null  hypothesis  for  this  test  is  that  observations  from  both 
groups  are  drawn  from  the  same  distribution,  and  therefore  Wa  should  equal  Wb.  Under  this  null 
hypothesis,  the  distribution  of  the  rank  sum  test  statistic  is  known  even  with  very  few 
observations  (Mann  &  Whitney,  1947). 83 


82 

This  explanation  of  the  Mann-Whitney-Wilcoxon  Rank  Sum  Test  is  based  on  several  textbooks  and  online 
resources  but  is  particularly  indebted  to  Wild  &  Seber  (1999). 


83 

With  many  observations,  a  normal  distribution  is  a  good  approximation  for  the  distribution  of  Wa  (Wild  &  Seber, 
1999). 
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Since  the  distribution  of  WA  under  the  null  hypothesis  is  known,  one  can  calculate  the 
likelihood  of  observing  a  rank  sum  as  or  more  extreme  than  WA  if  the  null  hypothesis  of  Wa=Wb 
is  true.  This  likelihood,  or  probability,  is  known  as  the  p- value.  As  used  here,  the  MWW  p-value 
gives  the  probability  of  observing  this  particular  distribution  of  patents  for  a  lab  if  the  lab  was 
actually  filing  patents  at  the  same  time  as  the  rest  of  the  field.  Errors  in  the  dataset  or  random 
fluctuations  in  when,  exactly,  the  lab’s  patent  attorney  submitted  the  paperwork  could  cause 
minor  shifts  in  the  reported  application  year  of  patents.  These  small,  random  shifts  could  make 
one  organization  appear  to  have  slightly  earlier  or  later  patent  applications  than  everyone  else.  If 
the  result  of  the  test,  the  p-value,  is  a  very  small  number,  then  it  would  be  highly  unlikely  that  the 
observed  timing  is  due  to  these  random  errors  alone.  Therefore,  small  p-values  increase  the 
confidence  that  the  observed  difference  in  timing  is  real.  The  traditional  thresholds  for  small  p- 
values  in  the  social  science  literature  are  0.1,  0.05,  or  0.01.  This  study  uses  a  less  conservative 
threshold  of  0.2,  which  is  sufficiently  large  to  consistently  identify  cases  that  have  a  large  visual 
separation  in  the  density  diagrams,  although  this  threshold  still  yields  a  variety  of  results  for 
visually  closer  cases. 

Both  one-sided  and  two-sided  tests  are  possible.  In  a  one-sided  test,  one  can  compare 
specifically  whether  the  lab’s  patent  distribution  is  shifted  either  to  the  left  (early)  or  the  right 
(late)  of  the  comparison  (i.e.  field)  distribution.  Formally,  in  a  one-sided  tests  the  alternate 
hypothesis  is  that  either  Wa<Wb  or  WA>WB,  and  the  p-value  is  the  area  under  the  left  or  right 
tail  of  the  distribution  respectively  (i.e.  from  WA  through  +/-  infinity).  In  the  two-sided  test,  one 
is  comparing  simply  whether  there  is  any  difference  in  the  timing  of  the  two  distributions. 
Formally,  in  the  two-sided  test  the  alternate  hypothesis  is  WA^WB,  and  the  p-value  is  the  area 
under  both  tails  of  the  distribution. 

MWW  test  example:  nanotechnology  innovators 

In  the  IBM  case  above  (figure  4.5),  the  two-sided  p-value  for  IBM’s  rank  sum  compared  with 
the  rest  of  the  field  is  4.7x  10"6.  For  the  one-sided  test  with  alternate  hypothesis  WiBm  <  Wuspto, 
the  p-value  is  2.4x  10"6.  Therefore,  it  is  highly  unlikely  that  one  would  see  this  degree  of  left¬ 
ward  (early)  shift  in  IBM’s  patent  trend  due  to  chance  alone. 

IBM’s  leadership  is  even  more  pronounced  when  compared  with  the  density  diagrams  of 
other  top  nanotechnology  patentees.  The  Navy  and  the  Massachusetts  Institute  of  Technology 
(MIT)  are  also  among  the  top  4  holders  of  nanotechnology  patents  in  the  U.S.  (H.  Chen,  Roco, 

Fi,  &  Fin,  2008).  Their  density  curves  versus  the  field  in  nanotechnology  class  977  are  shown 
below  in  figure  4.6.  In  the  density  diagram,  the  Navy’s  nanotechnology  trend  seems  to  have  the 
same  timing  as  the  field’s  curve.84  Except  for  a  2-patent  bump  in  the  1980s,85  MIT’s 
nanotechnology  trend  appears  to  lag  the  rest  of  field. 

84 

Note  that  NRL  accounts  for  the  majority  of  Navy  nanotechnology  patents  (Snow,  n.d.).  As  explained  in  section 
4.1.4,  NRL  has  a  small  swell  in  nanotechnology  as  a  result  of  a  combination  of  basic  research  grants  and 
discretionary  funding  supporting  more  applied  projects.  This  example  uses  all  patents  assigned  to  the  Navy  rather 
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For  the  Navy’s  nanotechnology  patent  portfolio,  the  p-value  for  the  2-sided  Rank  Sum  Test 
(alternate  hypothesis  is  that  the  Navy’s  patent  portfolio  is  shifted  either  left  or  right  from  the  rest 
of  the  USPTO  portfolio)  is  0.96.  Therefore,  under  the  null  hypothesis  (Navy’s  nanotechnology 
portfolio  has  the  same  timing  as  the  rest  of  the  field),  it  would  be  quite  likely  to  observe  this 
particular  portfolio  of  Navy  patents.  MIT’s  nanotechnology  patent  portfolio  shows  the  opposite 
trend.  The  p-value  for  the  two-sided  Rank  Sum  Test  for  MIT  versus  the  rest  of  the  field  is  0.17, 
indicating  that  it  is  somewhat  likely  that  MIT’s  patent  distribution  is  shifted  from  the  rest  of  the 
field.  The  one-sided  Rank  Sum  Test  with  alternate  hypothesis  of  MIT  >  USPTO  (i.e.  MIT’s 
patents  are  later)  has  a  p-value  of  0.085,  meaning  it  would  be  highly  unlikely  to  see  this 
distribution  of  patents  if  MIT’s  portfolio  actually  had  similar  timing  to  the  rest  of  the  field. 


Figure  4.6  Navy  (left)  and  MIT  (right)  nanotechnology  density  diagrams  versus  USPTO  (grey) 
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Summary  of  the  MWW  method  for  identifying  leadership 

The  MWW  Rank  Sum  Test,  as  applied  here,  identifies  the  likelihood  that  one  organization’s 
patent  pattern  is  shifted  earlier  or  later  than  the  pattern  of  patenting  in  the  field  as  a  whole.  This 
relative  timing  can  also  be  seen  by  comparing  an  organization’s  patent  density  curve  to  the 
density  curve  produced  by  all  patents  in  a  class.  Organizations  with  early  swells  are  innovators  in 


than  the  NRL  patents  identified  in  TechMatch  in  order  to  make  a  more  consistent  comparison  with  IBM  and  MIT’s 
patent  portfolios,  which  are  based  on  all  patents  (in  the  relevant  class)  assigned  to  those  organizations  in  the  USPTO 
2009  patent  grant  bibliographic  data  file. 

85 

These  two  patents  (US  patent  4,721,601  “Molecule-based  microelectronic  devices,”  filed  in  1984;  and  US  patent 
4,850,707  “Optical  pulse  particle  size  analyzer,”  filed  in  1986)  seem  unrelated  to  both  each  other  and  to  MIT’s  work 
after  the  take-off  point  based  on  a  review  of  the  relevant  patent  texts. 
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the  sense  that  they  were  among  the  first  to  produce  discoveries  and  accelerate  their  investment  in 
an  important  technology. 

Note  that  while  the  MWW  test  can  identify  spike  and  steady-work  trends  that  begin  before 
the  field  s-curve  takes  off,  this  discussion  only  compares  an  organization’s  swell  trend  to  the 
field  s-curve  trend.  This  restriction  simplifies  the  analysis  of  organizational  drivers  of 
innovation- via-leadership  which  follows  in  the  next  chapter.  Understanding  the  relationship 
between  spike  or  steady-work  trends  at  an  organization  and  the  rate  of  work  being  done  in  the 
rest  of  the  field  remains  an  area  for  future  studies. 

Nor  is  this  method  intended  to  provide  a  comprehensive  indicator  of  innovation.  As  with  all 
of  the  methods  suggested  in  this  dissertation,  it  should  be  used  as  part  of  a  suite  of  evaluation 
tools.  Specific  innovations  -  individual  inventions  that  are  novel  and  widely  used  -  may  also 
exist  within  spike,  steady-work,  or  lagging  swell  trends  and  may  be  identified  using  other 
methods. 

The  remainder  of  this  chapter  summarizes  the  frequency  of  leading  swells  in  the  DOD 
laboratory  portfolios  and  develops  hypotheses  to  explain  the  range  of  observed  results. 

4.2.3  Summary  of  leadership  results  for  DOD  labs 

Figure  4.8  (at  the  end  of  this  chapter)  summarizes  the  DOD  labs’  patent  portfolios  by  pattern 
type  and,  for  swells,  whether  that  trend  leads,  lags,  or  is  in-synch  with  the  rest  of  the  field.  Each 
row  in  figure  4.8  corresponds  with  the  data  for  one  DOD  lab.  The  labs  are  grouped  by  Service. 
Each  row  has  colored  boxes  indicating  classes  in  which  the  lab  has  at  least  6  patents  (i.e. 
marking  an  active  class  for  that  lab).  The  color  of  the  box  indicates  whether  that  lab’s  dominant 
pattern  in  that  class  is  a  swell,  a  spike,  or  steady-work. 

The  “+/-”  symbols  within  some  boxes  mark  swells  that  are  leading  or  lagging  the  field  in  that 
class.86  Swells  lead  the  field  if  the  p-value  in  the  fe/t-hand-sided  MWW  Rank  Sum  Test  (lab  has 
a  smaller  rank  sum  and  is  shifted  to  the  left)  is  less  than  0.2.  Swells  lag  the  field  if  the  p-value  in 
the  rz'g/7 /‘-hand-sided  MWW  Rank  Sum  Test  (lab  has  a  larger  rank  sum  and  is  shifted  to  the  right) 
is  less  than  0.2.  If  neither  condition  holds,  i.e.  the  p-value  is  greater  than  0.2  in  both  tests,  then 
the  swell  is  considered  in-synch  and  is  not  marked  with  a  “+/-”  symbol.87  Within  each  Service 
group,  the  rows  are  ordered  such  that  labs  with  the  most  leading  swells  are  at  the  top,  labs  with 
the  most  lagging  swells  are  in  the  middle,  and  labs  with  no  swells  are  at  the  bottom. 

This  being  a  new  theory  and  method,  there  is,  as  yet,  no  expected  rate  of  leadership  for  a 
high-performing  laboratory.  However,  leading  swells  are  rare  among  the  DOD  research  labs.  Of 


86 

The  MWW  Rank  Sum  Test  can  also  identify  spikes  and  steady- work  trends  that  lead  the  field.  However,  results 
for  spikes  and  steady- work  trends  are  not  shown  in  Figure  4.8  because  of  the  questions  raised  in  the  introduction  to 
section  4.2. 

87 

Note  that  for  these  comparisons  the  field  dataset  only  includes  patents  with  issue  dates  in  1990  or  later  in  order  to 
provide  a  fair  comparison  with  the  TechMatch  dataset.  Recall  the  TechMatch  dataset  includes  patents  issued  in  1990 
or  later  while  the  USPTO  dataset  includes  patents  issued  in  1975  or  later. 
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the  74  different  lab-class  trends  with  swells,  only  16  (22%)  lead  the  rest  of  the  field.  Only  7 
(28%)  of  the  24  laboratories  in  this  study  have  at  least  one  leading  swell.  Table  4.2  shows  the 
frequency  of  swells  and  leading  swells  relative  to  the  size  of  a  lab’s  portfolio  for  those  labs  with 
at  least  one  swell.  Leading  swells  make  up  less  than  one  quarter  of  any  single  DOD  lab’s 
portfolio,  and  the  upper  end  of  this  range  is  only  reached  in  a  lab  with  few  active  classes,  the 
Night  Vision  and  Electronic  Sensors  Directorate  (NVESD). 

Table  4.2  Frequency  of  leading  swells  in  DOD  patent  portfolios 


lab 

active 

classes 

swells 

leading 

swells 

leading 
swells  / 
active 
classes 

leading 
swells  / 
swell 
classes 

NVESD 

4 

1 

1 

25% 

100% 

NAWC  China  Lake 

20 

1 

1 

5% 

100% 

ARL 

53 

7 

5 

9% 

71% 

CERDEC 

12 

3 

2 

17% 

67% 

NRL 

54 

15 

4 

7% 

27% 

NSWC  Panama  City 

11 

4 

1 

9% 

25% 

SPAWAR  Systems  Center  Pacific 

27 

8 

2 

7% 

25% 

This  rarity,  perhaps,  should  be  expected.  Three  conditions  must  align  in  order  for  a  lab  to 
have  a  leading  swell  in  a  patent  class  in  this  study.  First,  the  field  s-curve  must  develop  in  the 
range  of  data  available  to  this  study:  between  1990  and  2010.  Second,  the  laboratory  itself  must 
have  a  swell.  Third,  the  laboratory  must  actually  be  ahead  of  the  field,  which  one  might  expect  to 
be  difficult  given  how  little  is  known  about  predicting  emerging  technologies. 

The  timing  of  the  development  of  the  field  s-curve  is  the  first  constraint  on  these  results.  If 
the  class  matured  prior  to  1990,  then  the  lab’s  own  work  will  appear  to  be  lagging,  or  at  best,  in¬ 
synch  because  the  TechMatch  dataset  does  not  go  back  far  enough  to  reveal  what  was  going  on 

OO 

at  the  lab  at  the  time  of  the  field’s  first  inflection  point.  If  the  class  has  yet  to  mature,  then  the 
lab’s  curve  will  also  seem  to  be  in-synch  or  lagging  because  the  proportion  of  field  patents  in  the 
left  tail  is  relatively  high.  See,  for  example,  ARDEC’s  work  in  Ammunition  and  Weapon  classes 
in  Figure  4.7.  While  ARDEC  has  a  swell  in  three  of  its  four  Ammunition  and  Weapon  portfolios 
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It  may  be  quite  difficult  to  compare  DOD  work  to  fields  that  matured  prior  to  1990  because  several  of  the  DOD 
laboratories  did  not  exist  in  their  current  form  prior  to  the  mid-1990s  (e.g.  see  ARL  (2003),  Duffner  (2000),  Hazell 
(2008),  Moye  (1997)).  Finding  patent  data  for  those  precursor  organizations  could  be  very  challenging  (see 
discussion  in  section  3.3). 


71 


(089,  102,  and  149),  because  the  rest  of  the  field  is  flat,  the  MWW  Rank  Sum  Test  finds  that 
ARDEC  is  lagging  (102,  149)  or  in  synch  (089). 

Figure  4.7  Lab  swells  versus  flat  field  curves 


It  is  plausible  that  many  fields  of  interest  to  this  study  would  have  peaked  long  before  1990. 
Mission  constrains  the  DOD  laboratories  to  work  in  certain  technology  areas,  and  those 
technology  areas  may  tend  to  be  old  and  already  mature.  For  example,  this  was  a  concern 
expressed  at  NVESD  which  conducts  research  into  optical  sensor  technologies  including  night 
vision  and  laser  designation.  The  U.S.  military  may  have  a  reputation  as  a  pioneer  in  night  vision 
and  infrared  sensing,  but  Army  research  in  that  field  dates  back  at  least  to  the  1950s  (NVESD, 
n.d.).  It  may  therefore  be  difficult  to  detect  leadership  in  NVESD’s  recent  patents.  However,  note 
that  this  study  did  detect  one  leading  swell  in  NVESD’s  patent  portfolio. 

In  addition,  note  that  Eusebi  (2011)  and  Eusebi  &  Silberglitt  (N.d.)  explored  s-curves  in  the 
patent  record  primarily  at  the  subclass  level  because  competing  trends  within  the  larger  primary 
classes  can  obscure  any  signal.  If  thousands  of  patents  are  being  filed  in  any  one  primary  class 
each  year,  then  a  ten  patent  uptick  in  one  subclass  would  be  drowned  out  by  the  overall  activity. 
Alternately,  declining  activity  in  one  subclass  or  natural  fluctuations  in  a  particularly  large 
subclass  can  cancel  out  any  increases  in  patenting  in  another  subclass.  Given  these  caveats,  it  is 
remarkable  that  this  study  found  s-curves  at  the  primary  class-level  at  all,  perhaps  indicating  how 
important  some  of  these  broad  technology  areas  have  become  to  modern  society. 

The  second  condition  that  must  hold  if  a  laboratory  is  to  have  a  leading  swell  is  that  not  only 
must  there  be  an  s-curve  in  the  field,  the  lab  must  also  have  a  swell  pattern  in  the  same  class. 
While  in  theory  one  could  compare  the  timing  of  a  lab’s  spike  or  steady- work  trend  to  the  takeoff 
point  of  the  field’s  s-curve,  the  interpretation  of  such  a  comparison  is  more  complex  (see 
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discussion  in  introduction  to  section  4.2).  One  must  therefore  be  careful  when  comparing  the 
leadership  rates  of  laboratories  with  few  or  no  swells,  like  ARL,  to  laboratories  with  many 
swells,  like  NRL.  Because  ARL  has  few  swells,  leading  swells  can  only  make  up  a  small  fraction 
of  its  total  active  class  trends.  The  fraction  of  swells  a  lab  has  that  are  leading  (right-most  column 
in  table  4.2)  may  be  a  more  appropriate  comparison  measure. 

Third,  if  there  is  an  s-curve  in  the  field  and  a  swell  at  the  lab,  the  lab  must  actually  be  ahead 
of  the  field,  naturally,  if  it  is  to  have  a  leading  swell  in  the  patent  record.  This  leadership  could  be 
by  design  or  by  chance.  However,  historically,  predicting  the  next  big  technology  has  not  had  a 
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high  success  rate.  For  example,  a  review  of  the  Army’s  early- 1990s  STAR  21  technology 
foresight  project  found  that  about  one-quarter  of  the  predictions  were  accurate  as  of  2008  and  the 
study  missed  a  number  of  key  trends  in  the  spread  of  the  internet  and  wireless  communication 
(Chait,  Valdes,  &  Lyons,  2009). 

It  may  be  that  labs  that  encourage  exploratory  research  have  more  leading  swells  out  of  sheer 
volume.  By  frequently  establishing  competencies  in  early-stage  technologies  such  labs  could  be 
in  a  better  position  than  less-exploratory  labs  to  capitalize  on  a  technology  when  it  takes  off. 

Such  labs  may  have  a  relatively  high  number  of  leading  swells  but  also  a  relatively  high  number 
of  in-synch  swells  -  reflecting  occasions  in  which  they  became  interested  in  an  early-stage 
technology  at  the  same  time  as  many  other  players  -  resulting  in  a  relatively  small  fraction  of 
leadership  among  their  swell  patterns.  Chapter  5  demonstrates  how  the  distribution  of  patterns 
within  organizations’  patent  portfolios  can  be  used  to  test  hypotheses  such  as  this  one  on  the 
practices  that  enhance  the  rate  of  innovation  in  research  organizations. 
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Arthur  C.  Clark  wrote,  “It  is  impossible  to  predict  the  future,  and  all  attempts  to  do  so  in  any  detail  appear 
ludicrous  within  a  very  few  years,”  and  furthermore,  “When  a  distinguished  but  elderly  scientist  states  that 
something  is  possible,  he  is  almost  certainly  right.  When  he  states  that  something  is  impossible,  he  is  very  probably 
wrong,”  (Clarke,  2000,  pp.  5,  1). 
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Figure  4.8  Summary  of  patent  patterns  in  DOD  labs 
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Chapter  5  Using  Patenting  Patterns  in  the  Study  of  Research  and 
Innovation  Processes 


The  introduction  to  this  dissertation  noted  that  two  major  barriers  in  science  policy  are  a 
limited  of  understanding  of  the  research  and  innovation  processes  and  the  lack  of  standard 
measures  with  which  to  study  these  processes.  Patenting  patterns  can  be  used  as  measures  in  the 
study  of  research  and  innovation  in  at  least  three  ways:  as  a  dependent  variable  in  project-level 
analyses,  as  a  classification  tool,  and  as  a  dependent  variable  in  organization-level  analyses. 

First,  the  type  of  pattern  associated  with  patents  from  a  research  project,  and  whether  that 
pattern  leads  or  lags  the  field,  could  be  used  as  an  indicator  of  research  output  quality. 

Comparing  and  contrasting  the  characteristics  of  research  projects  that  are  associated  with 
similar  patenting  patterns  can  help  isolate  project-level  factors  that  drive  the  quality  and  nature  of 
project  outputs. 

Second,  the  relative  fractions  of  each  pattern  in  a  laboratory’s  patent  portfolio  may  be  a 
classification  tool.  Laboratories  with  certain  research  approaches  or  missions  or  cultures  may 
tend  to  have  relatively  more  spikes  or  relatively  more  swells.  If  relationships  between  pattern 
fraction  and  certain  research  missions  or  cultures  can  be  established,  then  the  relative  number  of 
spikes  or  swells  in  a  lab’s  patent  portfolio  can  then  be  used  as  a  proxy  for  or  indicator  of  those 
characteristics  when  detailed  information  about  the  lab’s  character  is  not  available. 

Third,  the  fractions  of  spikes,  swells,  or  leading  swells  in  a  lab’s  portfolio  could  be  used  as 
measures  of  the  laboratory’s  outputs.  Comparing  across  laboratories  and  controlling  for  factors 
such  as  research  mission,  the  fraction  of  each  pattern  in  laboratory  portfolios  could  be  used  as  the 
independent  variable  in  an  analysis  of  factors  that  affect  the  generation  of  that  type  of  output.  A 
number  of  existing  hypotheses  about  the  research  an  innovation  process  might  be  reframed  in 
terms  of  quantifiable  laboratory  characteristics  such  as  pattern  fraction,  funding  levels,  staff 
demographic  characteristics,  etc.  Econometric  models  could  then  be  created  to  test  the 
hypothesized  relationships  between  these  factors. 

Chapter  4  discussed  the  first  application.  This  chapter  describes  the  second  and  third  uses  of 
patenting  patterns  and  sets  up  initial  examples  of  analyses  using  the  DOD  laboratories.  Section 

5.1  explains  the  basis  for  linking  pattern  fractions  with  laboratory  missions,  practices,  and 
cultures,  and  demonstrates  one  way  to  classify  laboratories  based  on  pattern  fractions.  Section 

5.2  demonstrates  the  use  of  pattern  fraction  as  the  independent  output  variable  using  a  simplified 
hypothesis  about  the  role  of  basic  research  in  establishing  new  lines  of  research. 
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5.1  Pattern  fraction  as  a  classification  tool 

As  discussed  in  section  4.1,  this  study  proposes  that  the  patenting  patterns  described  in  the 
previous  chapters  are  not  random  but  are  the  result  of  specific  behaviors  at  both  the  research 
group  and  the  organizational  level.  More  specifically,  steady-work  patterns,  this  study 
hypothesizes,  are  an  output  of  steady-work  research:  a  mode  of  work  in  which  the  lab  devotes  a 
steady  amount  of  resources  (e.g.  money,  people,  and  attention)  to  a  particular  field  of  study. 
Spikes  and  swells  represent  departures  from  this  baseline.  Spikes  are  instances  in  which  the 
laboratory  has  focused  more  resources  than  usual  on  producing  and  patenting  a  specific 
invention.  Swells  are  instances  in  which  the  lab  has  steadily  grown  its  investments  in  a  field  of 
research.  The  logical  framework  for  this  set  of  hypotheses  is  presented  in  section  4.1. 

If  the  pattern  of  patenting  a  lab  produces  within  a  single  technology  is  not  random,  then  the 
rate  at  which  spikes  and  swells  appear  in  their  overall  portfolio  may  not  be  random  either. 

Rather,  the  fraction  of  spikes,  swells,  and  other  patent  patterns  in  a  lab’s  portfolio  may  be 
directly  related  to  characteristics  of  the  organization.  Given  the  characterizations  of  the 
individual  patent  patterns  above,  labs  with  higher  fractions  of  swells  in  their  portfolios  could  be 
said  to  have  a  tendency  to  build  new  programs  out  of  their  initial  results,  while  labs  with  higher 
fractions  of  spikes  are  inventive  but  tend  to  move  on  to  new  research  areas  rather  than  continuing 
to  build  directly  on  past  discoveries. 

These  tendencies  could  be  a  direct  result  of  the  instructions  they  receive  from  their  parent 
organization,  i.e.  their  mission,  or  it  could  be  an  outcome  of  the  general  culture  of  the 
organization.  For  example,  laboratories  with  a  mission  to  create  specific  inventions  to  meet  the 
near-term  needs  of  a  parent  organization  (e.g.  a  military  service  or  a  commercial  parent)  may  be 
more  likely  to  have  many  spikes.  Meanwhile,  laboratories  with  more  general  missions  of 
scientific  discovery  may  be  more  likely  to  have  many  swells.  However,  a  laboratory  with  an 
invention-oriented  mission  but  a  culture  of  scientific  excellence  and  discovery  may  very  well 
have  many  swells  as  it  tries  to  increase  its  rate  of  innovation  by  building  on  past  breakthroughs. 

This  characterization  of  laboratories  as  creation/discovery  types  is  similar  to  the  Pasteur’s 
Quadrant  paradigm  established  by  Stokes  (1997).  Rather  than  thinking  of  basic  research  and 
applied  research  as  opposites,  Stokes  (1997)  suggests  a  two-dimensional  model  in  which  valuing 
advancing  the  boundaries  of  knowledge  (i.e.  discovery)  is  one  axis  and  valuing  the  utilization  of 
your  work  is  the  other  axis  (i.e.  creation).  Scientists  and  engineers  can  and  do  value  both 
simultaneously.  Stokes  describes  researchers  in  terms  of  the  quadrant  in  which  they  place  most 
value.  Researchers  who  place  a  high  value  on  discovery  and  a  low  value  on  creation  are  said  to 
be  in  Bohr’s  quadrant.  Researchers  who  place  a  high  value  on  creation  and  a  low  value  on 
discovery  are  said  to  be  in  Edison’s  quadrant.  Researchers  who  highly  value  both  are  said  to  be 
in  Pasteur’s  quadrant.  Returning  to  the  hypotheses  above,  labs  with  many  swells  may  be  in 
Bohr’s  quadrant,  while  labs  with  many  spikes  are  in  Edison’s  quadrant,  and  labs  with  equal 
numbers  of  spikes  and  swells  are  in  Pasteur’s  quadrant. 
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If  it  holds  that  spikes  and  swells  are  indicators  of  certain  types  of  research  behaviors,  then  the 
fraction  of  these  patterns  in  laboratories’  portfolios  can  be  used  to  classify  laboratories  into  sets 
with  like  behaviors.  Furthermore,  the  pattern  fraction  can  also  be  used  as  a  diagnostic  tool  to 
identify  best  practices  for  a  desired  research  outcome  or  as  a  monitoring  tool  to  assess  the  fit 
between  a  lab’s  desired  and  actual  performance.  Consider,  for  example,  how  the  DOD  labs 
cluster  in  terms  of  fraction  of  spikes  and  swells  in  their  overall  patent  portfolios. 

Figure  5.1  compares  the  fraction  of  steady  work,  swells,  and  spikes  in  each  DOD  lab’s 
portfolio.  While  a  number  of  clusters  are  possible,  the  labs  are  grouped  together  (black  outlines 
in  figure  5.1)  based  on  whether  their  portfolios  include  more  swells  than  spikes  (ARDEC, 
USAMRMC,  SPAWAR,  and  NUWC  Newport),  an  equal  number  of  swells  and  spikes  (NSWC 
Panama  City,  NSWC  Indian  Head,  NSRDEC,  NRL,  and  CERDEC),  or  more  spikes  than  swells 
(NSWC  Dahlgren  through  NAWC  Point  Mugu).  Within  each  group,  the  labs  are  listed  by 
increasing  fraction  of  steady  work.90 

The  bottom  five  labs  in  figure  5.1  are  not  included  in  a  cluster  because  of  their  low  numbers 
of  patents.  The  five  labs  each  have  65  or  fewer  patents  and  are  the  labs  with  the  smallest  patent 
portfolios  in  this  study.  Most  trends  at  these  labs  may  look  like  steady  work  because  the  labs 
rarely  patent  at  a  rate  high  enough  to  produce  a  spike  or  swell. 

If  the  hypothesis  stated  in  the  opening  of  this  chapter  is  roughly  correct  -  that  labs  with  an 
emphasis  on  creating  inventions  for  customers  produce  more  spikes  while  labs  with  a  focus  on 
general  scientific  discovery  produce  more  swells  -  then  one  would  expect  the  labs  in  the  top 
group  (more  swells)  to  have  a  relatively  stronger  discovery  focus,  the  labs  in  the  second  (even 
spikes  and  swells)  group  to  perhaps  have  some  emphasis  on  both  creation  and  discovery  ,  and  the 
labs  in  the  third  group  (more  spikes)  to  have  a  relatively  dominant  creation  focus. 

While  this  study  does  not  possess  sufficiently  detailed  information  about  the  characteristics 
of  each  DOD  lab  to  fully  evaluate  this  statement,  and  the  superficially  heterogeneity  of  the 
laboratories  in  each  group  in  figure  5.1  may  seem  to  cast  doubt  on  this  hypothesis,  there  is  some 
reason  to  hope  that  additional  information  on  the  character  of  the  DOD  labs  will  clarify  the 
extent  to  which  labs  do  or  do  not  fall  into  groups  along  the  creation/discovery  paradigm.  NRL 
and  ARL  do  appear  in  the  groups  one  would  expect  based  on  published  critiques  of  their 
activities.  NRL,  in  the  second,  equal  spike  and  swell  group,  could  be  described  as  a  Pasteur’s 
quadrant,  mixed  creation  and  discovery  lab.  ARL,  in  the  third,  more  spikes  group,  could  be 
described  as  an  Edison’s  quadrant,  creation-oriented  lab. 


90 

In  figure  5.1,  NAWC  Point  Mugu  is  out  of  order  within  its  group.  With  only  69  patents  and  3  active  classes, 
NAWC  Point  Mugu  may  be  better  placed  in  the  bottom  “no  patent”  group  than  with  the  “more  spikes  than  swells” 
group. 
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Figure  5.1  Distribution  of  patent  patterns  in  DOD  patent  portfolios 
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NRL  has  characteristics  of  both  a  creation- focused  and  a  discovery- focused  laboratory.  On 
the  creation  side,  Researchers  at  NRL  talk  about  being  “entrepreneurial”  (Personal  Interview, 
2012),  and  NRL’s  funding  process  encourages  researchers  to  find  projects  that  are  useful  to 
customers.  Around  80%  of  NRL’s  research  activities  are  customer-funded,  and  individual 
researchers  are  responsible  for  finding  customers  and  research  grants.  Even  in  the  20%  of  non¬ 
customer  work,  which  tends  to  be  more  basic  and  exploratory,  an  internal  review  process  rates 
proposals  based  on  both  scientific  merit  and  the  likelihood  that  the  results  will  transition  to  a 
customer  (Cozzens,  2001,  pp.  29-31).  Research  groups  that  consistently  fail  to  bring  in  work  that 
is  relevant  to  the  lab’s  mission  and  its  customers,  a  situation  known  at  NRL  as  “taking  in 
laundry”,  are  terminated  (Cozzens,  2001,  p.  31). 

On  the  discovery  side,  NRL  is  considered  by  some  to  be  the  DOD’s  top  basic  research 
laboratory  (Decker  et  al.,  2010,  p.  62).  Basic  research,  in  this  context,  is  research  that  is  directed 
at  expanding  the  frontiers  of  knowledge  rather  than  research  that  is  directed  toward  a  specific 
application  (DOD  regulation  7000. 14-R,  Volume  2B,  Chapter  5  (Office  of  the  Comptroller, 
2008)).  Furthermore,  Timothy  Coffee,  who  was  NRL’s  Director  from  1982  through  2001 
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(overlapping  in  time  with  the  first  half  of  this  data  set),  emphasized  connections  with  the  broader 
scientific  and  technical  community  as  a  core  competency  of  NRL’s  research  staff  (Chait,  2009). 

In  contrast,  ARL  has  been  described  as  more  near-term  applications  (i.e.  creation)  oriented 
than  desired  given  its  role  as  the  Army’s  corporate  research  lab  (Decker  et  al.,  2010). 

Researchers  at  ARL  talk  about  being  “mission-oriented,”  and  less  interested  in  exploratory 
research.  When  asked  about  how  they  conduct  exploratory  research  at  ARL,  one  interviewee 
responded  that  “sandboxing  is  discouraged”  (Personal  Interview,  20 12). 91  At  ARL,  most  work  is 
tied  to  Army  Technology  Objectives  (ATOs)  approved  through  a  high-level  review  process 
within  the  Army  S&T  system.  While  ARL  has  a  basic  research  portfolio,  most  of  its  in-house 
funding  is  applied  (see  section  B.2),  and  half  of  ARL’s  applied  work  is  tied  to  Army  needs 
through  the  ATOs  (DASA(R&T),  2010,  p.  I,  9-11). 

The  relative  fraction  of  spikes  and  swells  at  NRL  and  ARL  may  be  reflections  of  these 
different  operational  orientations.  NRL’s  entrepreneurial  creation  orientation  may  provide  an 
incentive  to  produce  spikes  in  many  technology  areas.  Meanwhile,  the  internal  emphasis  on 
continuing  excellence  in  basic  research  discovery  along  with  a  desire  to  build  unique 
competencies  that  enhance  the  lab’s  competitiveness  with  customers  may  lead  to  a  high 
frequency  of  swells.  However,  few  of  NRL’s  swells  are  leading  (4  of  15),  perhaps  reflecting  the 
difficulty  any  one  individual  has  in  predicting  the  next  important  research  area  and  in  convincing 
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a  customer  to  invest  in  an  unproven  field. 

At  ARL,  an  interest  in  producing  many  discrete  inventions  that  are  transferred  to  the 
warfighter  could  result  in  a  high  frequency  of  patent  spikes.  However,  if  the  goal  of  ARL’s 
research  staff  is  to  create  high-quality  inventions  in  the  technology  areas  identified  by  Army 
planners,  then  they  have  little  reason  to  search  out  work  in  emerging  and  un-prioritized 
technology  areas.  ARL  would  therefore  have  few  swells.  However,  given  the  high-rate  of 
leadership  among  ARL’s  few  swells  (5  of  7  are  leading),  when  they  do  pursue  more  exploratory 
projects,  ARL  and  Army  S&T  leadership  may  be  good  at  focusing  work  on  early-stage 
technologies  that  are  highly  likely  to  be  important  to  the  Army  in  the  future.  This  success  in 
technology  foresight  may  be  unrelated  to  the  factors  that  drive  ARL’s  overall  focus  on 
spikes/discrete  inventions. 

A  number  of  other  explanations  are  possible  for  the  relative  fraction  of  spikes  and  swells  at 
the  various  DOD  labs.  For  example,  medical  research,  which  is  the  focus  of  USAMRMC  (group 
1),  may  be  more  conducive  to  long  programmatic  research  arcs  than  missile  research,  which  is 
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Also  note  that  most  of  ARL’s  researchers  have  historically  been  civil  servants  with  low  turn-over,  who  cannot  be 
removed  for  “taking  in  laundry”  as  at  NRL,  but  the  lab  has  recently  grown  beyond  its  official  personnel  allocation 
and  has  brought  on  more  scientists  who  are  100%  funded  through  contracts  with  customers  (Personal  Interview, 
2012). 
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Recall  from  chapter  4  that  this  study  considers  spikes  to  be  indicators  of  potentially  high-value  inventions  while 
swells  are  indicators  of  growing  research  programs.  However,  in  addition,  swells  that  lead  a  field  s-curve  are 
considered  to  be  indicators  of  technology  leadership  on  the  part  of  the  lab.  This  leadership  may  coincidental  or  the 
result  of  a  deliberate  bet  on  the  part  of  the  laboratory  personnel. 
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the  focus  of  A  MR  DEC'  and  NAWC  China  Lake  (group  3).  Although,  under  this  alternate 
technology-type-based  explanation  it  remains  a  mystery  why  ARDEC  appears  to  behave  more 
similarly  to  a  medical  lab,  in  terms  of  patent  outputs,  than  to  other  ordnance  labs  (such  as  NSWC 
Dahlgren).93 

More  detailed  analyses  could  help  isolate  those  factors  that  do  consistently  explain  why 
laboratories  have  distinctly  different  patenting  pattern  outputs.  Furthermore,  if  such  analyses 
compared  laboratories  based  on  the  rate  at  which  they  produce  leading  swells,  then  such  analyses 
may  be  successful  in  isolating  significant  factors  affecting  innovative  research  performance. 
While  such  studies  could  be  qualitative,  along  the  lines  of  the  theory-based  discussion  above,  the 
following  section  provides  a  more  quantitative  example  of  this  type  of  analysis. 

5.2  Pattern  fraction  as  an  output  measure 

In  addition  to  its  utility  as  a  classification  tool,  the  fraction  of  each  pattern  in  an 
organization’s  patent  portfolio  could  be  used  as  measure  of  laboratory  outputs.  Controlling  for 
mission  and  other  characteristics  (per  the  above  discussion  not  all  laboratories  will  place  the 
same  emphasis  on  e.g.  spike-generating  research),  the  fraction  of  certain  pattern  types  in 
laboratory  patent  portfolios  could  be  used  as  the  dependent  variable  in  the  study  of  drivers  of 
certain  research  outputs. 

Given  the  breadth  of  the  science  and  innovation  management  literature,  there  may  be  a 
number  of  existing  hypotheses  regarding  drivers  of  research  performance  that  could  be 
interpreted  in  terms  of  pattern  fraction.  Patent  patterns  may  prove  an  easier  approach  to  measure 
these  types  of  outputs  than  surveys  or  other  current  methods.  To  illustrate  this  approach  to 
studying  research  processes,  this  section  demonstrates  a  simplified  analysis  comparing  two 
hypotheses  regarding  the  role  of  basic  research  in  establishing  new  lines  of  work  in  the 
ostensibly  application-oriented  DOD  laboratories. 

5.2.1  Sample  hypotheses 

Basic  research  is  thought  to  be  important  for  mission- focused  laboratories  because  it  provides 
a  way  to  engage  top  scientists  and  engineers  (e.g.  see  JASONs  (2009,  p.  1 1)),  promotes  a 
fundamental  approach  to  engineering  problem  solving  (see  Lyons  (2012)),  and  enables  the  lab  to 
develop  or  maintain  new  competencies  not  currently  covered  by  applied,  mission-oriented  work 
(e.g.  see  interview  comments  by  Lyons  and  Russo  in  Chait  (2009)).  In  terms  of  increasing  the 
number  of  swell  patenting  patterns  generated  by  a  lab,  basic  research  could  provide  the  base  of 
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Some  reviewers  have  asked  whether  the  number  of  active  classes  explains  the  fraction  of  spikes  or  swells  in  a 
lab’s  portfolio.  Labs  with  fewer  active  classes  and  narrower  technology  missions  (see  section  3.7.2)  could,  in  theory, 
have  more  swells  because  they  have  fewer  areas  in  which  to  invest  new  resources.  With  the  exception  of  those  labs 
that  have  so  few  patents  that  they  appear  at  the  bottom  of  figure  5.1,  this  seems  unlikely  to  be  the  case.  Preliminary 
analyses  conducted  for  the  work  reported  in  section  5.2  found  no  relationship  between  the  number  of  active  classes 
and  the  fraction  of  swells  or  spikes  in  a  DOD  lab’s  portfolio  using  a  variety  of  correlation  tests. 


80 


exploratory  work  that  starts  new  patent  swells.  Alternately,  an  active  basic  research  program 
could  impact  a  lab’s  patenting  rates  by  improving  the  efficiency  of  a  lab’s  applied  problem 
solving.  In  this  case,  larger  amounts  of  basic  research  may  be  related  to  more  patenting  swells 
only  in  as  much  as  that  basic  research  is  tied  to  patent-generating  applied  programs. 

The  remainder  of  this  section  (section  5.2.1)  expands  on  these  two  possible  connections 
between  basic  research  funding  and  patent  swells.  Section  5.2.2  constructs  measures  relevant  to 
each  of  the  two  hypotheses  using  data  from  the  1997-2007  DOD  In-House  S&T  Activities 
reports.  Section  5.2.3  discusses  the  results  of  a  simple  correlation  analysis  of  these.  Section  5.3.4 
closes  this  part  of  Chapter  5  by  summarizing  the  possible  utility  of  patenting  patterns  in  studying 
the  drivers  of  research  performance. 

Hypothesis  1 :  Basic  research  budgets  are  an  important  source  of  discretionary  exploratory 
research  funding  that  increases  the  number  of  patenting  swells 

Larger  amounts  of  basic  research  could  increase  the  rate  at  which  DOD  laboratories  identify, 
advance,  and  promote  promising  emerging  technologies.  For  example,  ARL’s  first  director,  Dr. 
Lyons,  noted  that  he  would  use  his  discretionary  funding  to  support  promising  early-stage 
research,  such  as  quantum  computing  in  the  late- 1990s,  which  was  not  yet  within  the 
technological  horizons  of  the  Army.  As  the  line  of  research  proved  itself,  it  could  be  transitioned 
to  an  Army  mission-supported  funding  stream  (Lyons,  2012,  pp.  4,  16).  If  basic  research  funding 
is  important  for  developing  research  programs  in  emerging  technologies,  then  DOD  research 
laboratories  with  larger  portions  of  their  funding  portfolios  designated  for  basic  research  may 
generate  more  swells.94 

Hypothesis  2:  In-house  connections  across  the  basic-applied-development  spectrum  are  an 
important  source  of  synergies  that  increase  the  number  of  patenting  swells 

Alternately,  expanding  on  the  role  of  basic  research  funding  in  enabling  fundamental 
approaches  to  applied  problem  solving,  basic  research  may  be  important  in  generating  swell 
patenting  patterns  only  in  as  much  as  that  basic  research  is  part  of  a  balanced  portfolio  of  basic, 
applied,  and  technology  development  work.  Large  amounts  of  basic  research  alone  may  not  lead 
to  increasing  rates  of  invention  and  patenting  because,  by  definition,  basic  research  is  not 
directed  at  any  particular  application.  Some  amount  of  applied  funding  may  be  necessary  to 
combine  a  basic  research  discovery  with  an  application  idea  such  that  the  concept  is  patentable. 
The  productivity  of  an  applied  research  portfolio,  however,  may  be  enhanced  -  perhaps  resulting 
in  more  patent  swell  patterns  -  if  it  is  paired  with  a  sufficient  amount  of  basic  research  from 
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They  may  also  generate  more  leading  or  in-synch  swells  since  they  are  more  able  to  target  early-stage 
technologies,  but  because  of  there  are  few  leading  swells  among  the  labs  in  this  study,  the  relationship  between  basic 
research  and  swell  leadership  will  not  be  tested  here. 
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which  it  can  draw  solutions.95  The  funding  measure  of  interest  in  this  second  hypothesis, 
therefore,  is  one  that  captures  the  relative  balance  in  a  laboratory’s  funding  among  basic,  applied, 
and  development  work. 

5.2.2  Constructing  funding  measures 

By  combining  funding  information  from  the  DOD  In-House  RDT&E  reports  with  the  patent 
pattern  results  presented  in  earlier  chapters,  this  study  can  explore  whether  there  is  evidence  of 
relationships  between  basic  research  funding  or  funding  balance  and  swell  generation. 

The  average  fraction  of  basic  research  in  the  labs’  1997-2007  in-house  S&T  budgets  is  used 
as  a  proxy  for  the  labs’  relative  emphases  on  basic  research.  Unfortunately,  budgets  for  the  entire 
range  of  patent  data  are  not  available.  Laboratory  reorganizations  and  changes  in  the  format  of 
the  DOD  In-House  RDT&E  funding  reports  make  it  very  difficult  to  identify  the  amount  of  in- 
house  basic  research  funding  provided  to  the  labs  in  this  study  prior  to  1997.  Fraction  of  in-house 
funding  dedicated  to  basic  research  is  used  instead  of  the  total  dollar  amount  to  account  for  the 
variation  in  the  absolute  size  of  budgets  across  labs.  Figure  5.2  compares  for  each  DOD  lab  in 
this  study  the  fraction  of  active  classes  that  are  classified  as  swells  against  the  average  fraction  of 
each  lab’s  in-house  S&T  budget  (1997-2007)  designated  for  basic  research. 

Again  using  the  average  funding  levels  between  1997  and  2007,  funding  balance  is  measured 
by  examining  the  variance  among  basic,  applied,  and  development  research  funding  proportions 
within  a  lab’s  in-house  research  budget.  Labs  with  more  evenly-divided  research  budgets  will 
have  smaller  funding  variances  and  hence  smaller  funding  balance  numbers.  For  example, 

NRL’s  in-house  research  funding  is  split  38%  basic,  38%  applied,  and  23%  development.  NRL’s 
funding  balance  measure  is  0.085,  which  is  the  standard  deviation  (square  root  of  the  variance) 
among  0.38,  0.38,  and  0.23.  ARL’s  in-house  research  funding  is  21%  basic,  78%  applied,  and 
less  than  1%  development.  ARL’s  funding  balance  measure  is  0.40.96  Figure  5.3  plots  the 
fraction  of  swells  in  each  lab’s  portfolio  against  a  measure  of  funding  balance. 
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The  histories  of  research  groups  are  replete  with  anecdotes  about  the  importance  of  collaborations  across  the 
basic-applied-development  spectrum  for  generating  innovations.  For  example,  the  history  of  Bell  Labs  in  Gertner 
(2012)  tells  of  several  occasions  in  which  applied  engineers  found  themselves  at  an  impasse  that  was  solved  when 
the  top  basic  research  scientists  in  the  organization  drew  on  their  expertise  in  mathematics  and  subatomic  physics  to 
suggest  design  changes.  Stokes  (1997)  describes  a  similar  synergy  between  practical  application  and  an 
understanding  of  principles  of  biology  and  chemistry  in  Louis  Pasteur’s  research  into  and  discovery  of  bacteria 
during  his  applied  attempt  to  standardize  beer  production. 

A  balanced  research  portfolio  could  also  synergistically  enhance  productivity  by  increasing  the  organization’s  ability 
to  use  a  wide  range  of  externally-generated  ideas.  Called,  absorptive  capacity,  it  is  thought  that  some  level  of 
internal  competence  in  a  field  is  necessary  before  an  organization  can  utilized  ideas  generated  elsewhere  (Cohen  & 
Levinthal,  1990).  Such  synergies  have  been  observed  in  industrial  R&D  organizations.  For  example,  Cassiman  & 
Valentini  (2009)  note  that,  while  technology  firms  have  increased  their  R&D  outsourcing,  they  have  also 
simultaneously  increased  the  fraction  of  their  in-house  research  portfolios  devoted  to  basic  research. 
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Credit  to  G.  Ryan  Faith  for  suggesting  this  summary  measure  of  funding  balance 
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Because  laboratory  mission  or  type  may  mediate  any  relationship  between  basic  research 
funding  and  generating  swells,  the  shape  of  the  points  in  figures  5.2  and  5.3  indicate  the  pattern 
group  to  which  the  lab  belongs.  While  a  number  of  proxies  for  research  mission  may  be  possible, 
for  consistency  with  section  5.1,  pattern  group  as  defined  in  figure  5.1  is  employed  as  an 
indicator  or  mission  or  lab  type  in  this  example.  Labs  with  more  swells  than  spikes  are  in  the  first 
cluster  (circle);  labs  with  equal  numbers  of  swells  and  spikes  are  in  the  second  cluster  (triangle); 
labs  with  more  spikes  than  swells  are  in  the  third  cluster  (square);  and  labs  with  few  patents  are 
listed  as  a  fourth  cluster  (plus). 

Note  that  because  of  how  data  is  reported  in  the  In-House  Activities  reports,  the  Naval 
Warfare  Center  labs  are  aggregated  into  their  parent  Warfare  Center  (NAWC,  NSWC,  NUWC, 
or  SPAWAR).  When  their  patent  pattern  results  are  aggregated,  the  NAWC  and  NSWC 
laboratories  fall  into  the  more  spikes  than  swells  group;  the  individual  labs  appear  in  all  but  the 
first  group  in  figure  5.1.  With  only  one  lab  with  sufficient  patent  data,  the  group  assignment  of 
NUWC  and  SPAWAR  remains  the  same  as  in  figure  5.1.  Also  note  that  funding  data  was  not 
available  for  the  Navy  Medical  Research  Center. 

Figure  5.2  Comparing  basic  research  funding  fraction  and  swell  fractions  of  DOD  laboratories 
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Most  of  the  data  points  in  figure  5.2  lie  in  a  vertical  band,  suggesting  that  overall  there  is  no 
direct  relationship  between  the  relative  intensity  of  basic  research  and  the  tendency  of  a  lab  to 
generate  swells.  Examining  the  data  by  group,  however,  suggests  that  there  may  be  an  inverse 
relationship  between  basic  research  fraction  and  swell  generation  among  labs  with  more  swells 
than  spikes  (circles). 
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Figure  5.3  Comparing  funding  balance  and  swell  fractions  of  DOD  laboratories 
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While  the  data  points  in  figure  5.3  are  still  scattered,  there  does  appear  to  be  a  moderate  trend 
running  from  the  top  left  to  the  bottom  right  of  the  displayed  range.  The  fraction  of  swells  in  a 
lab’s  patent  portfolio  seems  to  be  moderately  inversely  correlated  with  the  funding  balance 
measure  (more  evenly  distributed  portfolios  are  correlated  with  more  swells).  However,  when 
each  group  is  considered  separately,  the  existence  of  any  association  between  funding  balance 
and  swell  generation  becomes  less  clear. 

5.2.3  Testing  correlations 

In  order  to  estimate  the  likelihood  that  any  observed  trends  (or  lack  of  trend)  between  swell 
generation  and  the  two  funding  variables  exist  given  the  small  number  of  data  points,  table  5.1 
reports  the  Kendall  tau  rank  correlation  coefficient  and  the  p-value  for  the  null  hypothesis  of  no 
correlation  (tau  equals  0)  for  the  data  set  as  a  whole  and  for  each  group  separately.  Test  results 
for  the  association  between  basic  research  funding  fraction  and  swell  fraction  are  given  in  the 
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Kendall’s  tau  is  a  measure  of  correlation  that  examines  the  frequency  with  which  repeated  paired  observations  of 
two  random  variables  result  in  similar  rank  orderings  of  the  values  across  pairs  (Kendall,  1938).  A  simplified 
explanation  is  that  tau  measures  how  often  an  observation  with  a  relatively  large  value  for  variable  A  also  has  a 
large  value  for  variable  B.  Negative  values  of  tau  indicate  that  large  values  of  variable  A  tend  to  occur  with  small 
values  of  variable  B  or  vice  versa.  It  is  a  non-parametric  test  of  association  between  two  random  variables,  meaning 
that  the  likelihood  of  the  observed  association  can  be  tested  without  making  assumptions  about  the  underlying 
distributions  of  the  variables  or  the  shape  of  the  association  (a  reoccurring  requirement  in  this  dissertation).  When 
there  are  few  observations,  the  exact  likelihood  of  the  observed  Kendall’s  tau  can  be  calculated  by  enumerating  all 
possible  results  (e.g.  Valz  &  Thompson  (1994)).  This  study  uses  the  implementation  in  the  R  package  Kendall 
version  2.2. 
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middle  columns.  Results  for  the  association  between  funding  balance  and  swell  fraction  are 
shown  in  the  right-most  columns.  Values  of  tau  closer  to  +/- 1  indicate  a  stronger  direct/inverse 
(respectively)  association  between  two  variables.  The  p-values  in  table  5.1  indicate  the  likelihood 
of  observing  a  given  tau  if  there  is  actually  no  association  between  the  two  variables  (in  which 
case  tau  should  equal  zero),  i.e.  smaller  p-values  increase  the  confidence  that  the  observed  tau  is 
statistically  significantly  different  from  zero. 

Table  5.1  Kendall's  tau  and  significance  test  for  relationships  between  laboratory  funding 

variables  and  fraction  of  swells 


Group 
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p-value 
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p-value 

All 
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-0.7 

0.3 

0.0 
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0.3 

1.0 

-0.3 

1.0 
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7 

-0.0 

1.0 

0.0 

1.0 

The  Kendall’s  tau  tests  reported  in  table  5.1  support  the  statement  that  there  is  no 
relationships  between  the  fraction  of  basic  research  funding  and  the  fraction  of  swells  in  a  lab’s 
patent  portfolio  overall  in  the  data  set  or  for  the  swell-equal-spikes  or  the  swell-less-than-spike 
groups.  The  p-values  are  near  one  in  these  three  cases.  However,  there  may  be  a  strong  indirect 
relationship  between  basic  research  fraction  and  swell  fraction  in  labs  that  have  more  swells  than 
spikes  (tau  of  -0.7),  although  with  only  four  observations  the  p-value  of  the  significance  test  (0.3) 
is  outside  the  usually  accepted  upper  bounds  of  0.1,  0,05,  of  0,01.  In  addition,  the  relationship 
implied  by  the  location  of  the  circle  data  points  in  figure  5.3  is  in  the  opposite  direction  than 
expected  based  on  the  hypotheses  stated  above.  Overall,  the  results  comparing  basic  research 
budget  fraction  to  swell  generation  fraction  do  not  support  the  hypothesis  that  basic  research  is  a 
key  factor  in  starting  exploratory  research-driven  swells.  However,  given  the  small  number  of 
labs  in  this  preliminary  analysis,  it  may  be  worth  exploring  a  relationship  between  basic  research 
and  swell  generation  in  a  larger  data  set,  particularly  in  set  containing  more  swell-heavy  labs. 

In  the  correlation  tests  for  funding  balance  versus  swell  fraction,  none  of  the  results  are 
strictly  statistically  significant  at  traditional  levels.  In  any  individual  group,  the  Kendall’s  tau 
coefficient  between  funding  balance  and  swell  fraction  is  indistinguishable  from  zero.  However, 
the  weakly  inverse  relationship  (tau  of  -0.2)  between  funding  balance  and  swells  in  the  overall 
group  results  may  be  plausible  enough  (p-value  of  0.2),  given  the  limited  number  of  data  points, 
to  warrant  further  investigation. 

Keep  in  mind  that  the  p-value  gives  the  likelihood  of  a  test  statistic  as  or  more  extreme  as  the 
one  observed  if  the  null  hypothesis  of  no  relationship  holds.  While  a  20%  likelihood  may  not  be 
a  sufficient  level  of  certainty  for  policy  decisions,  given  the  limited  number  of  data  points  and 
the  number  of  uncertainties  involved  in  the  pattern  classification,  it  is  not  necessarily  a 
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sufficiently  level  of  doubt  to  strongly  rule  out  the  potential  of  further  research  to  uncover  some 
relationship. 

Note  that  in  the  analysis  of  funding  balance,  ARDEC  is  an  outlier,  and  as  in  the  discussion  of 
alternate  cluster  explanations,  something  of  a  mystery.  ARDEC  is  an  applied  research  lab  with 
no  appreciable  basic  research  funding  according  to  the  DOD  In-House  S&T  Activities  Reports. 
As  such,  its  funding  balance  measure  falls  in  the  middle -right  of  the  observed  range.  Yet  it  has 
the  highest  fraction  of  swells  in  its  portfolio  out  of  the  labs  examined.  It  may  be  that  there  is  in 
fact  no  relationship  between  funding  balance  and  swell  generation,  and  ARDEC’ s  location  in 
figure  5.3  is  within  the  natural  range  of  variation.  Alternately,  there  may  be  some  aspect  of 
ARDEC ’s  management  or  culture  that  results  in  more  swells  than  expected  based  on  the 
laboratory’s  funding  characteristics  and  technology  mission.  ARDECs  position  as  the  lab  with 
the  highest  fraction  swell  classes  in  this  study  alone  makes  it  a  priority  target  for  future  in  depth 
research. 

AFRL  is  also  an  outlier  in  that  is  has  a  more  balanced  funding  portfolio  but  few  swells. 
However,  the  AFRL  data  point  is  a  consolidation  of  all  of  the  AFRL  research  directorates 
because  of  the  limited  availability  of  Air  Force  patent  data  in  the  TechMatch  dataset.  Given  their 
size  and  history  as  independent  laboratories,  each  directorate  could  be  considered  a  research  lab 
equivalent  to  the  Army  RDECs.  As  such,  it  is  difficult  to  make  generalizations  about  AFRL’s 
research  behavior,  and  this  study  is  less  certain  about  AFRL’s  pattern  classification  result  than  in 
the  results  for  laboratories  with  more  complete  data. 

5.2.4  Summary  of  hypothesis  tests 

The  simplicity  of  the  model  and  the  limited  number  of  data  points  employed  preclude 
drawing  any  generalizable  conclusions  from  this  test,  but  the  results  could  be  interpreted,  at  a 
minimum,  to  emphasize  the  importance  of  considering  laboratory  type  when  searching  for 
drivers  of  research  performance.  Overall  among  the  DOD  laboratories,  the  relative  focus  on 
basic  research  at  the  lab  seems  unrelated  to  the  fraction  of  swells  in  the  lab’s  patent  portfolio. 
However,  basic  research  may  have  a  negative  role  in  swell  generation  in  labs  that  have  a 
particularly  preferential  orientation  toward  generating  swells  (i.e.  the  swell  >  spike  labs).  When 
looking  at  the  DOD  labs  overall,  balanced  funding  portfolios  seem  to  have  a  positive  association 
with  generating  swells  (more  balanced  labs  tend  to  have  more  swells),  but  this  may  be  a  result  of 
the  particular  characteristics  of  labs  with  swell-generating  research  missions.  When  the 
association  between  funding  balance  and  swells  is  analyzed  within  like-patenting  groups  (based 
on  fraction  of  patterns)  no  statistically  significant  association  is  found. 

There  are  a  number  of  possible  objections  to  this  analysis.  For  example,  the  funding 
measures  do  not  necessarily  cover  the  correct  time  periods,  and  several  of  the  laboratories  are 
actually  aggregations  of  different  facilities  (i.e.  AFRL  and  the  Naval  Warfare  Center  labs).  The 
intent,  however,  is  to  illustrate  the  potential  utility  of  patenting  patterns  for  operationalizing 
hypotheses  about  drivers  of  research  behavior  that  have  been  difficult  to  test  on  a  large  scale. 
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Future  work  will,  hopefully,  be  able  to  expand  on  this  line  of  work  with  more  organizations, 
more  control  variables,  and  more  complete  data  sets. 

5.3  Pattern  Applications  Summary 

The  patenting  patterns  identified  in  this  dissertation  have  potential  applications  in 
understanding  and  categorizing  the  behaviors  of  research  organizations,  identifying  the  roles  of 
different  factors  in  the  production  of  certain  research  outputs,  and  monitoring  the  performance  of 
research  organizations.  If  the  explanations  for  the  existence  of  patenting  patterns  presented  in 
Chapter  4  hold,  then  the  fractions  of  steady- work,  spike,  and  swell  patterns  in  an  organization’s 
patent  portfolio  provide  a  summary  measure  of  the  organization’s  research  and 
commercialization  activities.  Research  organizations  can  be  clustered  based  on  the  frequency  of 
these  patterns  in  their  patent  portfolios  (e.g.  figure  5.1),  and  additional  research  may  be  able  to 
identify  common  research  missions,  practices,  or  cultures  shared  by  laboratories  in  these  clusters. 

Furthermore,  it  may  be  possible  to  identify  specific  organizational  characteristics  or  research 
practices  that  tend  to  increase  the  production  of  certain  patenting  patterns  by  research 
organizations.  Combined  with  an  understanding  of  the  type  of  work  represented  by  each  pattern 
-  i.e.  spikes  as  potentially  high-value  discrete  inventions  and  leading  swells  as  innovative 
research  programs  that  lead  the  rest  of  the  field  -  such  analyses  could  help  identify 
characteristics  and  practices  that  improve  research  productivity  and  innovativeness.  For  example, 
the  analysis  of  relationships  between  funding  variables  and  the  production  of  swell  patenting 
patterns  in  section  5.2  demonstrates  ways  in  which  existing  theories  about  drivers  of  research 
performance  can  be  operationalized  and  tested  using  patent  and  budget  data. 

Patenting  patterns  can  also  be  used  in  a  monitoring  capacity  in  several  ways.  The  following 
bulleted  list  provides  a  few  examples. 

•  Once  links  between  patenting  patterns  and  types  of  research  behaviors  are  established,  the 
frequency  of  patent  patterns  can  be  used  to  compare  an  organization’s  actual  research 
behaviors  with  the  types  of  behaviors  preferred  by  the  organization’s  leadership. 

•  As  mentioned  in  sections  3.4  and  3.7,  a  lab’s  specific  active  patent  classes  (those  in  which  it 
has  at  least  6  patents)  can  be  used  as  one  proxy  of  the  lab’s  research  focus  areas.  These 
patent-based  focus  areas  can  be  compared  with  the  lab’s  stated  technology  mission  areas  to 
again  evaluate  the  fit  between  expectations  and  actual  performance. 

•  Program  staff  can  also  mine  patent  spikes  and  leading  s-curves  for  innovative  performance 
“nuggets”  -  examples  of  high-quality  research  outputs  from  the  lab. 

•  The  density  and  MWW  Test-based  patent  pattern  timing  comparisons  presented  in  section  4. 1 
can  also  be  used  to  compare  the  relative  timing  of  patent  activities  between  organizations. 
Such  comparisons  may  suggest  areas  in  which  technology  transfers  have  occurred  between 
laboratories,  since  labs  with  earlier  patents  are  potential  sources  of  knowledge  for  labs  with 
later  patents. 
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As  additional  research  clarifies  the  relationships  among  patenting  patterns,  innovative  and  other 
research  outputs,  and  research  organization  missions,  cultures,  and  practices,  the  list  of 
applications  could  grow  in  both  specificity  and  breadth. 
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Chapter  6  Summary 


Innovation,  particularly  government-driven,  is  an  increasingly  prominent  topic  in  public 
policy.  Innovation  and  its  role  in  supporting  the  U.S.  economy  is  now  a  regular  feature  in 
Presidential  speeches  and  legislative  proposals.  Yet  the  drivers  of  innovation  remain  poorly 
understood.  Studies  of  innovation  tend  to  produce  results  that  are  “multiple,  conflicting,  and 
vague,”  (see  the  review  in  van  der  Panne,  van  Beers,  &  Kleinknecht  (2003)). 

A  major  barrier  to  understanding  innovation  is  the  lack  of  good  performance  measures  and 
benchmarks  (Marburger,  2005).  Crow  &  Bozeman  (1998)  identifies  three  improvements  that  are 
needed  in  the  study  of  R&D  laboratories: 

•  “better  information; 

•  more  powerful  and  useful  concepts  and  classification  tools;  [and] 

•  a  willingness  to  use  information  in  analysis  and  evaluation.”  (p  6) 

This  dissertation  address  these  gaps  by  developing 

•  indicators  of  laboratory  inventiveness  based  on  the  patterns  of  an  organization’s  patenting 
over  time  rather  than  on  patent  networks  or  citation  rates; 

•  theories  regarding  the  types  of  research  activities  that  generate  these  patenting  patterns  and 
the  circumstances  in  which  the  patterns  denote  innovation;  and 

•  approaches  for  using  the  relative  frequency  of  each  patenting  pattern  in  an  organization’s 
portfolio  to  analyze  possible  drivers  of  research  organization  performance  and  evaluate  the  fit 
between  an  organization’s  expected  and  actual  research  behaviors. 

The  following  sections  summarize  each  contribution. 

6.1  Patenting  Patterns 

Patents  have  long  been  used  as  a  performance  measure  for  research  programs  and 
organizations.  In  addition  to  being  indicators  of  general  knowledge  production,  patents  are  also 
potential  indicators  of  innovation.  Patents  are  by  definition  related  to  inventions  that  are  novel 
and  potentially  useful,  key  aspects  of  innovations.  However,  the  actual  innovativeness  of  a  given 
patent  can  lie  anywhere  within  a  very  large  range  of  values,  and  existing  estimates  of  patent 
value  may  not  be  appropriate  for  detecting  innovative  results  of  public  research.  While  citation 
volume  has  been  frequently  used  as  a  proxy  for  patent  value,  this  study  contributes  to  a  line  of 
research  that  is  investigating  whether  relationships  in  the  patent  technology  classification  system 
provide  better  insights  into  the  pursuit  of  innovations  (e.g.  Eusebi  (2011;  N.d.)  and  Strumsky  et 
al.  (2012)). 
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S-curves  have  been  used  to  describe  the  rate  of  performance  improvement  of  a  technology, 
the  diffusion  of  ideas,  and  the  rise  of  substitute  products  in  a  market.  S-curves  also  appear  in  the 
patent  record  as  inventors  collectively  decide  to  bet  on  a  new  technology  (Eusebi  &  Silberglitt, 
n.d.).  These  collective  bets  grow  the  number  of  patents  in  a  technology  area  (as  represented  by 
the  patent  technology  class/subclass)  at  an  accelerating  rate  until  both  patent  examiners’ 
willingness  to  accept  the  novelty  of  new  claims  (Eusebi,  2012)  and  the  marketplace  for  the 
technology  become  saturated.  Inventors  and  organizations  who  tend  to  patent  in  the  early  parts  of 
s-curves  are  more  innovative  than  others  in  the  sense  that  they  have  pioneered  a  new  technology 
area  that  later  became  a  popular  subject  for  investment  and  research  (Eusebi,  2011). 

While  an  organization’s  position  in  the  overall  technology  s-curve  is  one  indicator  of 
innovativeness,  examining  the  organization’s  pattern  of  patenting  over  time  can  reveal  more 
information  about  that  organization’s  history  of  activity  in  a  technology  area.  Histograms  of  an 
individual  organization’s  patents  by  application  date  and  technology  class  demonstrate  four  basic 
patterns  of  patenting  at  the  organizational  level  (see  figure  3.8):" 

1 .  Steady- work:  A  single  organization  will  generally  have  only  a  few  patents  in  any  given 
technology  class.  Even  in  an  organization’s  larger  classes,  the  most  common  filing 
pattern  is  a  handful  of  patents  every  few  years.  This  pattern  of  activity  appears  as  a  series 
of  level  data  points  in  the  organization’s  histogram  of  patents  per  year  per  class. 

2.  Spikes:  Occasionally  an  isolated  burst  of  patents  will  appear  above  the  level  of  steady 
work.  A  group  of  patents  qualifies  as  a  spike  when  three  or  more  patents  are  filed  within 
two  years  by  the  same  group  of  inventors  in  closely  related  technology  areas.  Spike  cases 
examined  thus  far  have  been  the  result  of  successful  research  projects  that  ultimately 
transferred  technology  to  a  military  or  commercial  user. 

3.  Swells:  Rather  than  one  sudden  burst  of  patenting,  the  third  pattern  appears  as  a  gradual 
increase  in  the  number  of  patents  filed  by  an  organization  in  a  class  each  year.  This  trend 
can  be  thought  of  as  a  reflection  of  an  organization’s  increased  investment  in  or 
efficiency  of  research  in  this  technology  area. 

4.  Submarines:  When  there  is  a  long  delay  between  the  filing  and  issuance  of  a  patent,  a 
cluster  of  patents  may  appear  in  an  organization’s  portfolio  far  removed  in  time  from  the 
rest  of  their  work.  These  patents  are  called  submarines  because  of  the  long  period  of  time 
during  which  they  are  unpublished  and  hidden  from  view.  When  a  submarine  surfaces,  it 
can  disrupt  existing  players  in  a  technology  by  triggering  negotiations  between  the  patent 
owner  and  existing  producers  of  the  technology  who  were  unaware  of  the  hidden  patent. 
Prior  to  changes  in  application  rules  in  2001,  (patent  applications  are  now  published  18 
months  after  they  are  submitted),  some  organizations  used  application  revisions  to 


99 


Excluding  the  case  of  no  patents  in  a  technology  class 
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deliberately  create  submarines  (Graham  &  Mowrey,  2004).  Today  similar  patterns  can 
still  result  from  patent  secrecy  orders  requested  by  the  government. 

6.2  Patenting  Patterns  as  Indicators  of  Innovation 

The  presence  of  spikes  and  swells  in  a  laboratory’s  patent  portfolio  can  identify  areas  in 
which  the  laboratory  has  been  potentially  innovative.  Furthermore,  the  relative  timing  of  a 
laboratory’s  swell  trend  compared  with  a  similar  increase  in  patenting  in  the  field  as  a  whole  can 
be  an  indicator  of  innovation  via  technology  leadership. 

Spikes  signal  a  potentially  innovative  output  from  research  organizations  in  the  sense  that 
they  represent  inventions  which  the  organization  expects  to  be  highly  valuable.  This 
interpretation  follows  from  two  lines  of  reasoning.  First,  from  a  purely  financial  standpoint, 
patents  cost  the  filing  organization  significant  sums  of  money.  Patent  spikes  represent  clusters  of 
related  inventions  in  which  the  laboratory  has  made  a  relatively  large  investment  in  patent 
protection  compared  with  inventions  protected  by  only  a  single  patent.  Second,  clusters  of 
patents  on  related  inventions  (aka  fencing  or  surrounding)  are  thought  to  be  a  more  effective 
defensive  strategy  than  single  patents.  Patent  spikes,  therefore,  put  an  organization  in  a  better 
position  to  defend  their  claims  from  potential  competitors  than  would  a  single  patent  alone. 

However,  the  methods  presented  in  this  dissertation  alone  do  not  provide  direct  evidence  that 
the  spikes  are  actually  used.  A  number  of  factors  could  limit  the  actual  use  of  what  the  lab 
thought  was  a  promising  invention.  For  example,  the  laboratory  could  misjudge  the  demand  for 
its  invention  or  the  invention  could  prove  more  difficult  to  implement  than  originally  thought. 
Therefore,  while  it  is  tempting  to  call  spikes  signals  of  innovations,  additional  work,  such  as  a 
case  study,  is  necessary  to  confirm  actual  use  and  actual  innovation. 

On  their  own,  swells  reflect  capability  building,  but  the  relative  timing  of  an  organization’s 
swell  relative  to  the  rest  of  the  field  may  indicate  innovation.  Swells  that  lead  the  rest  of  the  field 
denote  occasions  in  which  the  organization  has  innovated  by  being  among  the  first  to  do  work  in 
a  popular  technology  area.  Leadership  can  be  detected  visually  by  overlaying  the  organization 
and  the  field  patent  density  curves  on  the  same  time  axis,  or  it  can  be  detected  by  applying  the 
Mann- Whitney- Wilcoxon  (MWW)  Rank  Sum  Test  to  compare  timing  shifts  between  the 
respective  patent  series. 

6.3  Studying  Research  Performance  via  Patenting  Patterns 

The  overall  distribution  of  patenting  patterns  in  a  lab’s  portfolio  may  be  an  indicator  of  the 
lab’s  overall  research  mission,  culture,  and  practices.  For  example,  labs  with  many  spikes  but 
few  swells  may  be  labs  that  focus  on  producing  many  discrete  inventions  but  have  relatively  less 
interest  in  contributing  to  the  advancement  of  scientific  knowledge.  Labs  with  more  swells  than 
spikes  may  place  more  emphasis  on  accumulating  scientific  knowledge  and  place  less  emphasis 
on  transitioning  inventions  to  users.  Labs  with  similar  rates  of  spikes  and  swells  in  their  patent 
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portfolios  may  be  operating  in  what  Stokes  (1997)  called  “Pasteur’s  quadrant”  -  a  mode  of 
research  in  which  new  scientific  knowledge  is  pursued  in  order  to  create  more  innovative  applied 
products.100  While  additional  research  is  still  needed  to  more  fully  explain  how  patenting  patterns 
are  related  to  research  behaviors,  comparing  and  contrasting  the  characteristics  of  research 
organizations  with  similar  and  different  fractions  of  patterns  in  their  portfolios  can  likely 
improve  the  understanding  of  drivers  of  research  performance  (for  example  see  figure  5.1  for  a 
classification  of  the  DOD  laboratories  by  patenting  patterns). 

In  addition,  more  specific  quantitative  analyses  of  relationships  between  the  distribution  of 
patenting  patterns  and  specific  laboratory  characteristics  -  such  as  the  amount  of  basic  or  applied 
research  funding  or  the  percent  of  the  staff  with  PhDs  -  can  help  evaluate  the  importance  of 
these  characteristics  for  producing  particular  types  of  research  outputs.  For  example,  if  basic 
research  is  important  for  generating  swells  because  it  is  a  source  of  foundational  ideas  and 
exploratory  funding,  then,  ceteris  paribus,  labs  with  larger  emphases  on  basic  research  should 
also  have  relatively  more  swells  in  their  patent  portfolios.  A  preliminary  analysis  of  the 
associations  between  research  funding  and  patenting  patterns  (presented  in  section  5.2)  is 
inconclusive  but  suggests  that  any  relationships  are  likely  mediated  by  the  laboratory’s  mission 
or  character,  which  in  turn  reemphasizes  the  importance  of  understanding  the  extent  to  which 
patenting  patterns  are  indicators  of  labs’  overall  research  missions,  cultures,  and  practices. 

6.4  Limitations  and  Future  Research 

The  analyses  presented  in  this  dissertation  employ  a  list  of  DOD  laboratory-generated  patents 
obtained  from  the  DOD  TechMatch  website.  The  TechMatch  list,  as  accessed  in  January  2012, 
includes  an  incomplete  sample  of  DOD  laboratory  patents  issued  between  1990  and  2011.  While 
several  statistical  techniques  have  been  used  to  account  for  the  uncertainties  introduced  by  an 
incomplete  data  set,  repeating  the  analyses  presented  in  this  dissertation  over  a  longer  and  more 
complete  data  set  would  improve  confidence  in  both  the  pattern  classification  rules  used  and  the 
specific  pattern  classification  results  for  the  organizations  examined  in  this  study. 

While  expanding  the  range  of  data  on  which  this  study’s  findings  are  based  would  increase 
confidence  in  the  pattern  classification  results,  additional  qualitative  research  would  help  refine 
the  interpretation  of  these  results.  Comparing  the  histories  and  outcomes  of  swell  (leading, 
lagging,  and  in-synch),  spike,  and  steady-work  projects  would  help  refine  understanding  of  the 
types  of  research  activities  for  which  these  patenting  patterns  are  signatures.  A  large-scale  survey 

100  Rather  than  thinking  of  basic  research  and  applied  research  as  opposites,  Stokes  (1997)  suggests  a  two- 
dimensional  model  in  which  valuing  advancing  the  boundaries  of  knowledge  (i.e.  discovery)  is  one  axis 
and  valuing  the  utilization  of  your  work  is  the  other  axis  (i.e.  creation).  Scientists  and  engineers  can  and 
do  value  both  simultaneously.  Stokes  describes  researchers  in  terms  of  the  quadrant  in  which  they  place 
most  value.  Researchers  who  place  a  high  value  on  discovery  and  a  low  value  on  creation  are  said  to  be  in 
Bohr’s  quadrant.  Researchers  who  place  a  high  value  on  creation  and  a  low  value  on  discovery  are  said  to 
be  in  Edison’s  quadrant.  Researchers  who  highly  value  both  are  said  to  be  in  Pasteur’s  quadrant. 
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of  inventors  by  pattern  type,  which  obtained  narratives  of  the  project  history  and 
commercialization  attempts,  would  provide  a  body  of  evidence  for  such  research. 

Even  as  this  line  of  research  moves  forward,  managers  of  government  labs  can  use  the  patent 
patterns  described  in  this  study  to  evaluate  and  monitor  their  organization’s  performance.  For 
example,  patenting  patterns  can  be  used  to 

1 .  Answer  “what  have  you  done  for  me  lately”  by  highlighting  research  projects  that 
potentially  transitioned  to  users  (e.g.  spikes  and  leading- swells) 

2.  Monitor  laboratories’  core  competencies  and  shifts  in  those  competencies  over  time  by 
tracking  changes  in  the  rate  of  patenting  in  laboratories’  active  patent  classes  over  time 

3.  Identify  possible  areas  of  supporting  or  duplicative  overlap  in  research  activities  by 
comparing  the  timing  of  patenting  activities  in  different  laboratories  within  the  same 
technology  classes,  and 

4.  Conduct  open-source  intelligence  by  applying  these  tools  to  the  patent  portfolios  of  other 
science  and  technology  organizations  foreign  and  domestic 

Furthermore,  while  the  research  presented  in  this  dissertation  is  exploratory,  it  is  a  step 
towards  establishing  an  objective,  relevant,  and  tolerably  easy  to  use  indicator  of  research 
quality.  By  comparing  patenting  patterns  within  and  across  organizations,  future  research  will 
not  only  refine  the  interpretation  of  within-organization  patenting  patterns  but  can  also  refine  the 
understanding  of  drivers  of  research  performance.  In  particular,  the  spike  and  leading  swell 
patterns  are  potentially  indicators  of  innovative  research  programs.  As  such,  future  research  that 
identifies  drivers  of  spike  and  leading  swell  patenting  patterns  also,  potentially,  identifies  drivers 
of  innovation.  Hopefully,  an  improved  understanding  of  within-organization  patenting  patterns 
will  lead  to  improved  science  and  innovation  policy  making  as  well. 
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Appendix  A  Common  Research  Evaluation  Methods 


Program  evaluation  of  government  research  is  an  evolving  field  (Ruegg  &  Jordan,  2007). 
Ruegg  &  Jordan  (2007)  lists  13  methods  for  measuring  the  performance  of  R&D  programs,  each 
tailored  to  a  different  question.  Ruegg  &  Feller  (2003)  also  provides  an  excellent  summary  of  the 
pros  and  cons  of  using  a  variety  of  qualitative  and  quantitative  evaluation  methods  on  science 
and  technology  programs.  Table  A.l  summarizes  common  R&D  program  evaluation  methods, 
their  uses,  and  their  limitations  based  on  these  sources. 

Many  of  the  methods  listed  in  Table  A.l  are  targeted  toward  on-going,  internal  management 
of  R&D  programs  rather  than  evaluation  by  outside  researchers.  For  measuring  the 
innovativeness  of  DOD  labs,  many  fail  either  the  relevance,  ease  of  use,  or  credibility  criteria 
from  Parish  (1998): 

•  Bibliometric  indicators  such  as  journal  publication  and  citation  counts  are  a  standard 
measure  of  scientific  output,  but  are  not  necessarily  relevant  to  the  production  of  military¬ 
relevant  knowledge  and  hardware. 

•  Peer  review  is  the  standard  approach  to  evaluating  research  organizations  with  difficult  to 
observe  outcomes,  but  by  definition  is  potentially  biased  by  the  perceptions  of  the  experts 
doing  the  review  and  the  anecdotes  provided  by  the  program  under  review. 

•  Benchmarking  compares  the  laboratory’s  inputs  and  process  to  those  believed  to  be  found 
in  high-performing  research  organizations.  However,  many  questions  remain  regarding 
which  inputs  and  processes  are  really  related  to  various  outcomes.  For  example,  the 
National  Academies'  Laboratory  Assessment  Board  recommends  monitoring  the  quality 
of  the  workforce  on  the  theory  that  "Many  organizations  have  determined  that  the  quality 
of  the  workforce  is  the  most  reliable  predictor  of  future  R&D  performance,"  (Laboratory 
Assessments  Board,  2012,  p.  4).  However,  this  recommendation  runs  counter  to  the 
findings  of  J.  T.  Hall  &  Dixon  (1975)  which  found  that  simple  inputs,  such  as  staff 
quantity  and  quality,  were  poor  predictors  of  outcomes  such  as  the  transfer  of  knowledge 
to  users. 

•  General  indicators  are  similar  to  benchmarking  but  less  explicitly  tied  to  theories  of 
“good”  research  environments. 

•  Technology  commercialization  studies,  benefit-cost  studies,  and  spillover  studies  are 
difficult  to  use  in  a  military  setting  because  the  primary  benefit,  military  performance,  is 
difficult  to  monetize  and  because  the  path  from  laboratory  to  commercialization  or 
deployment  can  be  extremely  difficult  to  trace. 

•  Network  analysis  could  help  trace  those  paths,  and  could  be  helpful  in  evaluating  some  of 
the  training  and  communication  goals  of  the  DOD  laboratories,  but  on  its  own  it  does  not 
really  address  the  impact  of  technological  discoveries. 

•  Historical  tracing  methods  do  address  that  impact  as  well  as  knowledge  flows,  but  they 
are  extremely  resource  intensive.  Project  Hindsight,  which  traced  the  contributing 
research  to  20  significant  military  inventions,  consumed  approximately  40  man-years  of 
research  time  (Kreilkamp,  1971). 
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•  Econometric  studies  could  be  very  useful,  but  the  appropriate  data  and  models  are  not 
always  clear. 

•  Similarly,  the  appropriateness  of  general  case  study  methods  and  survey  methods 
depends  on  the  broader  framework  of  the  analysis.  For  example,  this  study  uses  case 
studies  to  provide  additional  context  for  patent  data. 

In  addition,  note  that  quantitative  research  impact  measures  may  be  collected  by  a  laboratory 
but  not  reported  due  to  fears  about  how  that  data  will  be  used.  Kostoff,  Averch,  &  Chubin  (1994) 
acknowledge  that  the  results  of  a  research  impact  assessment  may  result  in  the  cancellation  of  a 
"pet"  project  or  lead  to  budget  cuts  from  disgruntled  legislators.  ARL  reportedly  collects  a 
number  of  statistics  about  its  research  enterprise  including  papers  and  patents,  percentage  of 
doctoral  level  staff  members,  numbers  of  visiting  scientists,  overhead  rates,  and  workforce 
turnover,  but  they  are  for  internal  use  only  (Cozzens,  2001). 


Table  A.1  Summary  of  laboratory  evaluation  methods 


Method 

Description 

Typical  Uses 

Primary  Challenges 

Peer  Review/Expert 
Judgment 

Formal  or  informal  review  of 
programs  and  projects  by  individuals 
who  are  experts  in  that  area  of 
research.  Usually  involves 
presentations  by  researchers 
followed  by  discussion  by  the 
peer/expert  panel 

Project  selection,  In¬ 
progress  review  of  quality, 
Assessment  of  portfolio 
balance  and  relevance 

Subjective  -  variation 
among  reviews,  potential 
for  reviewer  bias 

Monitoring,  Data 
Compilation,  and 

Use  of  "Indicators" 

On-going  tracking  of  inputs,  outputs, 
and  outcomes  including  money  spent, 
publications,  technological  objectives 
achieved 

In-progress  review, 
Assessment  of  portfolio 
quality 

Selecting  appropriate 
metrics  and  obtaining  data 

Bibliometric 

Methods:  Counts 
and  Citation  Analysis 

Counting  academic  publications, 
academic  citations,  or  patents 

Provide  evidence  of 
knowledge  creation  and 
dissemination 

Citation  and  publication 
rates  vary  across  fields, 
Does  not  account  for 
informal  or  non-academic 
knowledge  dissemination 

Bibliometric 

Methods:  Data 

Mining 

Text  analysis  to  identify  relationships 
within  varied  documents  including 
technical  reports,  academic 
publications,  resumes,  and  press 
releases 

Provide  evidence  of 
knowledge  creation  and 
dissemination,  Identify 
research  gaps  and 
opportunities 

Large  data  gathering  and 
cleaning  burden  (newer 
tools  provide  more 
automation),  Development 
of  visualization  and 
interpretation  tools  is  on¬ 
going 

Benchmarking 

Comparison  with  other  programs 

Determine  relative  quality 
of  program,  Identify  best 
practices 

Identifying  relevant 
comparison  programs  and 
measures,  Obtaining  data 
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Method 


Technology 
Commercialization 
Tracking  Method 


Benefit-Cost  Case 
Study 


Description 


Tracking  performance,  cost,  and 
market  share  data  of  products 
derived  from  the  research  program 


Quantifying  positive  and  negative 
impacts  of  a  project  or  program  and 
comparing  them  to  the  project  or 
program  costs 


Typical  Uses 


Identify  (applied)  program 
impact,  Support  benefit- 
cost  studies 


Evaluate  (applied) 
program  impact  relative  to 
costs,  Estimate  potential 
impact  of  proposed 
projects 


Primary  Challenges 


Tracing  &  verifying 
technology  or  knowledge 
transfer,  Separating 
effects  of  R&D  program 
from  technology  transfer 
efforts 

Attributing  impacts  to  the 
research  project, 
quantifying  qualitative  or 
subjective  impacts 


Spillover  Analysis 
Using  a  Combination 
of  Methods 


Network  Analysis 


Historical  Tracing 
Method 


Econometric 

Methods 


Estimating  the  public  and  private  Provide  a  broader 
value  of  spillovers  from  research  such  estimate  of  program 
as  the  spread  of  new  knowledge  or  impacts 
benefits  of  new  technology  that  are 
not  fully  incorporated  in  the  price  of 
that  technology 


Visualize  networks  of  researchers 
based  on  co-authorship,  participation 
in  research  centers,  email  traffic, 
reported  face-to-face  contacts,  etc. 


Working  forward  from  a  discovery  or 
backward  from  an  impact  identify  all 
subsequent/contributing 
(respectively)  discoveries 


Using  mathematical  models  and 
statistical  tools  to  explore 
relationships  between  program  goals 
and  actual  (quantifiable)  impacts 


General  Case  Study  Use  narratives  supported  by  other 
Methods  data  to  explore  program  processes 

and  impacts 


Provide  evidence  of 
knowledge  creation  and 
dissemination,  Evaluate 
impact  of  program  on 
communication  and 
collaboration  networks, 
Identify  relevant 
researchers 

Identify  role  of  program  in 
known  innovations 


Evaluate  program  impacts, 
provide  support  for  cause- 
and-effect  relationships, 
hypothesis  testing 


Explore  or  evaluate 
program  processes, 
Provide  illustrative 
examples 


Identifying  and  attributing 
spillovers,  see  Counts  and 
Citation  Analysis,  Network 
Analysis,  and  Econometric 
Methods 


Large  data  gathering  and 
cleaning  burden, 
Accounting  for  impacts 
and  changes  overtime 
increases  difficulty 


Identifying  all  contributions 
can  require  a  large 
number  of  interviews,  a 
large  amount  of  data 
mining,  and  skillful 
"detective  work"  (Ruegg  & 
Jordan,  2007) 

Availability  of  data  and 
appropriate  models,  Large 
data  gathering  and 
cleaning  burden,  Applying 
and  interpreting  complex 
statistical  methods 

Generalizability,  Access  to 


General  Survey  Ask  program  managers,  participants,  Determine  participant  or 
Methods  or  other  stakeholders  questions  about  stakeholder  satisfaction  or 

program  processes,  outputs,  and  needs,  Gather  data  to 
outcomes  support  other  methods 


program  processes,  outputs,  and  needs,  Gather  data  to  respons 

outcomes  support  other  methods  Comply 

e.g.  identify  outputs  and  Federal 
outcomes  not  recorded  Reducti 

elsewhere 

Sources:  (G.  Jordan  &  Malone,  2001;  Ruegg,  2007;  Ruegg  &  Feller,  2003;  Ruegg  &  Jordan,  2007) 


Skillful  survey  design, 
Obtaining  adequate 
response  rates, 
Complying  with  the 
Federal  Paperwork 
Reduction  Act 
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Appendix  B  Review  of  Patent  Value  Studies 


The  main  difficulty  for  using  patents  as  a  proxy  for  innovation  lies  in  confirming  use. 
Demburg  and  Gharrity  identify  three  components  to  patent  use:  "1)  the  patented  invention  is 
actually  exploited  commercially,  2)  the  patent  is  of  value  to  its  owner  for  purposes  of  building  up 
a  dominant  market  position  or  for  bargaining  for  cross-licenses,  etc.,  [and]  3)  the  patent  is 
considered  potentially  useful  because  of  the  possibility  that  either  the  invention  or  the  patent  will 
be  used  in  the  future,"  (Demburg  &  Gharrity,  1961,  pp.  358-359).  Demburg  and  Gharrity's  first 
aspect  of  use  is  most  relevant  to  this  study.  However,  direct  evidence  of  a  resulting  product  can 
be  hard  to  find.  Most  studies  fall  back  on  some  estimate  of  a  patent's  "value"  which  captures 
multiple  connotations  of  "use". 

There  are  two  ways  to  approach  patent  valuation:  directly  and  indirectly.  Direct  measures  of 
patent  value  try  to  attribute  a  dollar  value  to  individual  patents  or  an  entire  patent  portfolio. 

Direct  measures  are  usually  obtained  through  surveys,  some  of  which  ask  about  patent  use 
explicitly,  or  through  models  that  relate  a  firm's  knowledge  assets  to  its  market  value.  Indirect 
measures  rank  patents  based  on  some  other  characteristic  that  theoretically  correlates  with  value, 
use,  or  some  other  concept  of  innovativeness.  A  patent's  history  of  renewal  payments  is  an 
indirect  measure  of  value  that  has  been  in  use  since  at  least  the  1930s.  Perhaps  the  most  common 
indirect  measure  in  use  today  is  the  number  of  citations  a  patent  receives  from  other  later  patents 
(a.k.a.  forward  citations).  This  section  describes  the  results  other  researchers  have  obtained  using 
direct  and  indirect  methods,  explains  how  well  each  method  provides  evidence  of  use  or 
innovativeness,  and  states  some  of  the  challenges  associated  with  using  each  method  in  a 
government  setting. 

B.1  Direct  measures 

Inventor  surveys 

Surveys  that  ask  inventors  whether  their  patent  was  used  to  produce  a  product  that  was  sold 
are  clearly  direct  measures  of  whether  a  patent  was  used  to  produce  a  product  that  was  sold.  A 
1950s  survey  of  600  patent  assignees  found  that  over  50%  of  patents  owned  by  corporations 
were  used  at  some  point  in  their  lifetime  (Sanders,  Rossman,  &  Harris,  1958)  (also  see 
discussion  of  the  Patent  and  Trademarks  Foundation  study  in  Schmookler  (1966,  pp.  48-50)  and 
Griliches  (1990)).  Between  40  and  50  percent  of  inventor-owned  patents  were  also  eventually 
used  (Schmookler,  1966,  p.  52).  Surveys  that  ask  inventors  to  judge  the  value  of  their  patents, 
however,  measure  something  slightly  different. 
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A  patent's  value  and  whether  it  was  used  should  be  correlated,  although  they  are  distinct 
concepts.  On  average,  patents  that  are  used  to  produce  a  commercial  product  should  have  a 
higher  value  than  those  that  are  not  tied  to  a  product,  but  patents  that  are  not  used  will  not 
necessarily  have  zero  value.  In  addition  to  actual  product  revenues,  the  value  an  inventor  assigns 
to  their  patent  may  be  based  on  licensing  revenues  (which  do  not  necessarily  result  in  a  marketed 
product)101  or  the  perceived  value  of  their  technology  portfolio  for  attracting  investors  or 
negotiating  other  deals. 

Inventors  and  patent  owners  may  also  value  patents  because  of  their  role  in  a  larger  business 
strategy.  A  patent  represents  a  right  to  exclude  others  from  making  or  using  a  technology  (see  the 
discussion  of  the  decision  to  patent  in  Murphy,  Orcutt,  &  Remus  (2012)).  If  a  patent  holder 
declines  to  use  the  technology  and  also  excludes  others  from  using  the  technology,  then  the 
technology  effectively  does  not  exist.  That  exclusion,  however,  still  has  value  to  the  patent 
owner. 

Supporting  this  distinction  between  use  and  value,  the  survey  by  the  Patent  and  Trademarks 
Foundation  found  that  1 1%  of  patents  "though  not  used,  have  had  or  are  expected  to  have  other 
benefits,"  (Sanders  et  al.,  1958).  A  further  7%  of  patents  were  licensed  but  not  reported  as  "used" 
by  their  original  assignee  (Sanders  et  al.,  1958). 

More  recent  surveys  have  also  found  that  around  50%  of  patents  are  used  to  support  a 
commercial  product.  While,  according  to  Griliches  (1990),  as  of  1990  the  Patent  and  Trademarks 
Foundation  survey  was  the  most  extensive  survey  of  patent  use  ever  conducted,  in  the  2000s 
several  large  surveys  were  done  of  international  patent  holders.  In  addition,  four  smaller  U.S.  and 
European  surveys  were  executed  in  the  early  2000s  to  understand  the  relationship  between 
patenting  and  scientific  information  sources  in  large  corporations  (Giuri  et  al.  (2007)  reviews 
these  studies).  The  two  recent  large-scale  and  broadly-focused  surveys  are  discussed  below. 

The  Patval-EU  survey,  conducted  in  2003  and  2004,  received  responses  from  the  inventors  of 
over  nine-thousand  patents  granted  by  the  European  patent  office  between  1993  and  1997.  The 
goal  of  the  Patval-EU  survey  was  to  both  continue  to  explore  the  validity  of  other  indicators  of 
patent  value,  particularly  citation-based  measures,  and  provide  data  on  inventor  characteristics 
and  the  research  process  on  a  larger-scale  than  had  been  done  before.  Patval-EU’s  measure  of 
patent  value  was  the  amount  for  which  inventors  would  have  been  willing  to  sell  their  patent 
immediately  after  it  was  granted  given  what  they  currently  knew  about  the  benefits  to  holding 
the  patent.  Patval-EU  also  asked  inventors  to  indicate  how  their  patent  was  used.  In  a  result  that 
is  consistent  with  other  studies,  they  found  that  slightly  more  than  50%  of  patents  were  used  to 
protect  a  commercial  product  or  were  licensed  to  another  firm.  Almost  20%  of  patents  in  the 
study  were  held  to  block  other  firms  (Giuri  et  al.,  2007). 


101  Licensed  patents  do  not  get  used  for  many  practical  reasons.  Several  of  these  reasons  boil  down  to:  the  invention 
isn't  as  profitable  as  expected.  Maybe  the  market  is  smaller  than  expected.  Maybe  the  invention  is  more  difficult  to 
manufacture  and  therefore  costs  more  than  expected.  Federico  (1937)  has  a  larger  discussion  of  this  topic. 
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The  Research  Institute  of  Economy,  Trade  and  Industry  (RIETI)-Georgia  Tech  inventor 
survey,  conducted  in  2007,  sampled  over  5000  Japanese  and  triadic  patents  (patents  that  had 
been  submitted  to  the  European,  U.S.,  and  Japanese  patent  offices)  filed  in  1995  (Sadao  & 

Walsh,  2009;  Suzuki,  2011).  The  RIETI-Georgia  Tech  study  borrowed  from  the  Patval-EU  study 
when  designing  its  questionnaire.  The  main  purpose  of  the  survey  was  to  understand  how 
industrial  research  processes  differed  between  Japan  and  the  U.S.,  but  the  survey  asked  inventors 
to  indicate  the  value  of  their  patent  in  one  of  four  bins  and  indicate  whether  the  patent  had  been 
used  within  the  company,  licensed  to  another  party,  or  used  by  a  spin-off  company  (Suzuki, 
2011).  The  RIETI-Georgia  Tech  study  found  that  60%  of  the  patents  in  their  sample  had  been 
used,  although  Japanese  firms  were  more  likely  to  use  the  patents  to  support  products  within  the 
inventing  firm  than  license  the  patents  to  another  firm  or  use  them  for  a  spin-off  company  (Sadao 
&  Walsh,  2009).  Interestingly,  among  the  surveyed  triadic  patents,  those  requiring  no  R&D  or 
those  which  were  generated  from  targeted  research  programs  were  the  most  likely  to  be  in  the 
highest  value  bin  (as  opposed  to  patents  that  resulted  from  a  serendipitous  discovery  or  general 
research)  (Nagaoka  &  Walsh,  2009). 

Patent  valuation  via  stock  price 

Rather  than  ask  a  firm  about  the  dollar  value  of  their  patent  portfolio,  some  researchers  will 
try  to  estimate  patent  value  based  on  changes  in  a  firm’s  stock  market  value.  This  approach  is  not 
applicable  to  government  laboratories  since  they  do  not  offer  stock  or  otherwise  have  an  overall 
monetary  value  like  a  private  company.  However,  this  stock-based  valuation  approach  is 
reviewed  below  for  completeness. 

Estimating  the  change  in  stock  price  attributable  to  a  company's  patent  portfolio  is  often  done 
to  compare  the  R&D-driven  competitiveness  of  different  companies.  Figure  3  in  Griliches  (1990) 
illustrates  the  thought  process.  A  company’s  overall  profitability  or  stock  market  value  is  a 
function  of  both  its  physical  assets  and  its  non-physical  knowledge  and  human  capital  assets. 
Patents  are  one  proxy  for  unobservable  research  knowledge.  Using  the  known  value  of  the 
company,  the  known  estimate  of  the  replacement  value  of  physical  assets,  and  the  known  size  of 
the  company’s  patent  portfolio,102  researchers  can  use  regression  to  estimate  the  fraction  of  a 
firm’s  value  attributable  to  its  patent  portfolio.  Since  the  size  of  the  patent  portfolio  is  a  proxy  for 
the  firm’s  knowledge  assets,  researchers  interpret  this  coefficient  as  a  measure  of  the  quality  of 
the  firm’s  R&D  and  other  knowledge  processes. 

The  main  question  here  with  respect  to  patents  is  whether  they  are  a  sufficiently  good  proxy 
for  knowledge  to  be  used  in  its  place  in  equations  describing  productivity  improvements.  R&D 
expenditures  seem  to  be  a  bigger  factor  in  determining  the  stock  price  than  the  number  of  patents 
a  firm  has  (Griliches,  1990).  Pakes  (1985)  uses  a  more  complex  model  to  try  to  understand  the 


102 

Sometimes  the  patent  variables  are  time-lagged,  depreciated,  or  weighted  by  technology  area  or  by  citation 
counts.  B.  H.  Hall,  Mairesse,  &  Mohnen  (2010)  review  commonly  used  lag  and  depreciation  coefficients. 
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relationships  among  a  firm’s  R&D  expenditures,  the  size  and  quality  of  its  patent  portfolio,  and 
other  events  that  change  the  way  the  market  values  a  firm’s  knowledge  assets.  While  Pakes  is 
able  to  identify  differences  between  short-term  shocks  that  increase  a  firm’s  stock  market  value 
and  the  long-term  market  value  due  to  the  firm’s  general  level  of  R&D  activity,  he  seems 
uncertain  about  the  direction  of  causality  in  the  shocks.  Did  a  new  research  discovery,  with  its 
associated  patent,  cause  the  market  to  reevaluate  the  firm’s  value  or  did  some  other  event  cause 
the  market  to  place  more  weight  on  a  firm’s  knowledge  assets  causing  the  firm’s  managers  to 
increase  their  R&D  and  patenting  activities?  Balasubramanian  &  Sivadasan  (201 1)  find  that,  at 
least  for  first  time  patentees,  patents  affect  firm  value  primarily  by  increasing  new  product  sales, 
although  they  too  are  unwilling  to  make  a  strong  causal  statement,  noting  that  there  are  a  number 
of  external  factors  that  could  affect  the  decision  to  patent,  new  product  sales,  and  firm  value 
simultaneously. 

Drawbacks  to  survey  methods 

There  are  two  drawbacks  to  using  survey  methods  to  understand  the  use  of  patents  for 
military  technologies.  The  first  and  main  drawback  to  survey-based  methods  of  patent  valuation 
is  time.  Each  survey  is  a  major  investment  in  resources  requiring  determining  the  sample  frame, 
gathering  contact  information,  developing  the  survey  instrument,  and  following  up  with  target 
respondents.  Furthermore,  the  process  must  be  repeated  for  each  new  population  (both  in  terms 
of  laboratory  and  time  frame)  in  the  study.  This  makes  surveys  expensive  to  use  as  an  on-going 
monitoring  tool. 

The  second  drawback  is  related  to  the  different  motivations  and  mechanisms  for  patenting  in 
the  DOD  laboratories  versus  a  private  company.  Because  government  laboratories  have  different 
motivations  for  patenting  than  private  companies  and  inventors,  asking  inventors  and  technology 
transfer  managers  about  the  value  of  patents  may  have  a  different  meaning  than  it  does  in  the 
commercial  sector.  In  addition,  inventors  and  research  managers  may  not  be  able  to  say  whether 
a  patent  was  used  or  what  its  approximate  value  is.  Therefore,  survey  questions  used  in  a 
commercial  setting  may  not  work  well  in  a  government  laboratory. 

Inventors,  because  they  are  bureaucratically  separated  from  the  technology  transfer  and 
acquisition  organizations  may  not  know  if  their  patent  has  been  used  in  a  commercial  license 
(although  they  should  receive  royalty  payments  if  the  license  included  a  payment  to  the 
laboratory).  Conversely,  while  inventors  often  are  aware  when  their  research  was  transferred  to  a 
military  system,  they  may  not  know  which  particular  patents  were  tied  to  that  invention.103  The 
laboratory’s  Office  of  Research  Technology  Applications  (ORTA)  does  keep  a  record  of  all 
licenses  from  the  lab’s  patents,  but  they  are  not  always  at  liberty  to  say  whether  or  not  a 
particular  patent  has  been  licensed  (Personal  Interview,  2012). 


103  This  problem  was  encountered  in  several  early  interviews. 
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Furthermore,  in  a  military  setting,  counting  licensing  revenue  alone  can  undervalue  patents 
that  are  used  for  military  rather  than  commercial  purposes.  If  a  government  researcher  invents 
and  patents  something,  and  then  that  invention  is  used  for  government  purposes,  for  example  in  a 
military  system,  then  no  money  may  change  hands.  Government  technology  transfer  offices  say 
that  the  government  can  negotiate  more  favorable  purchase  terms  if  the  government  owns  the 
patent  rather  than  the  contractor.  In  theory,  if  the  government  has  already  developed  a 
technology  embodied  in  a  patent,  then  they  would  be  able  to  pay  the  contractor  just  for 
manufacturing  and  not  for  research  and  development.  However,  none  of  the  technology  transfer 
personnel  interviewed  could  identify  an  example  of  this  actually  happening.104  Even  if  it  did 
occur,  researchers  outside  the  government  would  likely  not  have  access  to  the  information 
necessary  to  calculate  the  amount  of  money  saved,  i.e.  the  difference  between  the  contract  price 
with  the  patent  and  without  the  patent. 

Overall,  value  in  a  government  laboratory  may  be  evaluated  very  differently  than  in  a  private 
company  or  in  a  university.  While  commercial  value  is  considered  because  it  generates  income 
for  the  lab  and  for  the  inventor,  potential  licensing  revenue  is  definitely  not  the  only  or  primary 
concern  for  DOD  technology  transfer  offices.  DOD  technology  transfer  and  patent  professionals 
generally  listed  defensive  patenting  (obtaining  a  favorable  contract  as  described  above)  as  the 
first  motivation,  although  they  acknowledged  that  actually  defending  their  patent  space  was  very 
difficult  (Personal  Interview,  2012). 

Another  DOD  motivation  for  securing  patents  is  encouraging  the  development  of  a  key 
industry.  By  offering  non-exclusive  licenses  to  multiple  suppliers,  DOD  labs  can  push 
technologies  they  believe  to  be  better  into  the  defense  industrial  base.  One  example  of  this 
mechanism  was  given  to  us  by  a  Navy  patent  professional.  A  particular  pipe  coating  was 
developed  at  a  Navy  lab  and  licensed  to  numerous  ship-building  subcontractors.  The  coating 
eventually  became  the  new  industry  standard.  The  value  of  patents  used  for  this  mechanism 
would  also  be  difficult  to  obtain  from  a  simple  survey  of  laboratory  patent  offices  (Personal 
Interview,  2012). 


104  One  manager  was  actively  derisive  of  the  idea,  noting  that  several  different  organizations,  who  rarely  coordinate, 
would  be  involved  in  enforcing  government  patent  rights.  In  the  military,  the  laboratory  Offices  of  Research  and 
Technology  Applications  (ORTA)  track  government-owned  patents,  but  they  are  mostly  focused  outward  on 
technology  transfer  to  non-military  uses.  The  acquisition  officers  that  manage  government-purpose  procurement  are 
in  an  entirely  different  organization  and  have  no  knowledge  of  patents  created  at  and  owned  by  the  laboratories. 
Even  if  the  laboratory  told  the  acquisition  office  about  a  specific  patent,  acquisition  contracts  for  complex  military 
systems  are  written  in  terms  of  performance  specifications  rather  than  technology  specifications.  Therefore  the 
acquisition  officer  may  not  be  able  to  tell  the  contractor  to  use  the  technology  represented  by  the  specific  patent. 
Furthermore,  if  the  military  somehow  later  realized  that  they  had  paid  extra  money  for  technologies  that  they  had 
created,  they  would  have  to  go  to  the  Justice  Department  to  file  a  lawsuit  (Personal  Interview,  2012).  One 
government  patent  attorney  interviewed  during  this  study  mentioned  that  it  is  the  job  of  the  laboratory  patent 
attorneys  to  help  negotiate  favorable  terms  for  contracts  including  government  intellectual  property,  but  the  patent 
attorneys’  time  can  become  oversubscribed  with  requests  for  new  patent  applications  instead  (Personal  Interview, 
2012). 
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B.2  Indirect  measures 


In  contrast  to  survey  methods  described  above,  methods  that  value  patents  based  on  some 
recorded  and  archived  piece  of  data  may  have  a  large  initial  investment,  but  once  the  algorithm  is 
implemented,  applying  the  code  to  a  new  dataset  can  be  trivial  (depending  on  the  complexity  of 
data  cleaning  required  to  make  new  raw  data  usable).  There  are  two  common  indirect  methods 
for  estimating  patent  value,  both  of  which  take  advantage  of  archival  data:  patent  renewal 
payments  and  forward  citation  counts. 

Patent  renewal  payments 

In  addition  to  the  fees  an  inventor  has  to  pay  when  they  submit  a  patent,  several  patent 
systems  require  periodic  payments  to  keep  the  patent  active.  Using  renewal  payments  to  estimate 
patent  value  is  one  of  the  oldest  approaches.  Early  studies  in  this  area  were  done  on  European 
patent  systems  (see  for  example  Federico  (1937)  and  Demburg  &  Gharrity  (1961)).  The  U.S. 
only  started  requiring  renewal  payments  in  1980  (Griliches,  1990).  The  current  USPTO  renewal, 
or  maintenance,  fees  are  $1,130  after  3.5  years,  $2,850  after  7.5  years,  and  $4,730  after  1 1.5 
years.105 

Patent  renewal  payments  only  approximate  use  for  several  reasons  including  because  they  are 
based  on  an  inventor's  perceived  value  of  the  patent  and  because  the  signal  is  truncated  at  the 
upper  end.  Being  renewed  may  be  a  good  indicator  that  a  patent  was  used  since  inventors  likely 
only  renew  patents  from  which  they  are  deriving  some  commercial  benefit.  However,  there  are  a 
number  of  reasons  to  suspect  that  renewal  data  tells  us  very  little  about  actual  use. 

First,  as  discussed  above,  the  benefit  an  inventor  bases  his  renewal  decision  on  may  not  be 
directly  tied  to  an  actual  product.  For  example,  if  the  inventor  licensed  the  patent  to  someone 
who  was  unsuccessful  in  bringing  the  invention  to  market,  or  if  the  patent  was  part  of  some 
larger  signaling  strategy.  Furthermore,  inventors  may  make  the  first  (or  first  few)  renewal 
payments  still  with  the  hope  of  some  future  return. 

Second,  renewal  payments  are  a  coarse  measure  of  value  because  they  lump  together  patents 
that  may  have  very  dissimilar  outcomes  into  a  few  bins.  In  the  U.S.  system  there  are  only  four 
possible  bins:  (1)  never  renewed,  (2)  renewed  once,  (3)  renewed  twice,  and  (4)  renewed  three 
times.  Patents  in  the  first  category  may  include  both  trivial,  "wacky"  patents  and  "early 
bloomers",  patents  that  generate  substantial  returns  in  the  first  few  years  but  become  obsolete  in 
later  renewal  periods  (Gambardella,  Harhoff,  &  Verspagen,  2008).  On  the  late  end,  Sanders 
argues  that  patents  that  survive  all  three  renewal  periods  almost  certainly  saw  some  use  (see 
attachment  to  Demburg  &  Gharrity  (1961)),  but  two  different  patents  that  both  survive  all  three 
renewal  periods  may  still  generate  very  different  commercial  returns.  Sanders  also  points  out  that 


see  the  USPTO  fee  information  website  for  updates  to  the  fee  schedule 
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returns  generated  by  a  patent  associated  with  a  commercial  product  are  so  much  larger  than  the 
renewal  fees  that  the  renewal  decision  is  "trivial". 

Third,  lapsed  patents  are  not  necessarily  unused  since,  as  Frederico  puts  it,  lapsed  patents 
become  "part  of  the  common  fund  of  useful  technology"  and  may  be  employed  by  anyone 
(Federico,  1937).  Organizations  whose  primary  interest  in  patenting  is  to  prevent  another 
organization  from  exerting  monopoly  control  over  the  technology,  like  the  DOD  labs  dealing 
with  military  contractors,  may  allow  patents  to  lapse  early  in  order  to  protect  their  interests  with 
the  minimum  expenditure.  However,  in  the  current  U.S.  patent  system,  publishing  the  discovery 
in  an  academic  journal  or  trade  magazine  would  be  a  more  cost  effective  way  to  place  the 
invention  in  the  public  domain  (Personal  Interview,  2012). 

Forward  citations 

Forward  citations,  the  citations  a  patent  receives  from  other  later  patents,  are  an  attractive 
measure  of  patent  value  because  they  suggest  relationships  between  foundational  technologies 
and  their  children.  In  his  study  comparing  patent  forward  citation  rates  with  commercial  product 
value,  Trajtenberg  cites  as  justification  for  his  approach  the  description  of  forward  citations  made 
by  the  U.S.  Patent  Office’s  Office  of  Technology  Assessment  and  Forecast: 

“During  the  examination  process,  the  examiner  searches  the  pertinent  portion  of 
the  "classified"  patent  file.  His  purpose  is  to  identify  any  prior  disclosures  of 
technology. . .  which  might  anticipate  the  claimed  invention  and  preclude  the 
issuance  of  a  patent;  which  might  be  similar  to  the  claimed  invention  and  limit 
the  scope  of  patent  protection. . .;  or  which,  generally,  reveal  the  state  of  the 
technology  to  which  the  invention  is  directed.  If  such  documents  are  found  they 
are  made  known  to  the  inventor  and  are  "cited"  in  any  patent  which  matures  from 
the  application. . .  Thus,  the  number  of  times  a  patent  document  is  cited  may  be  a 
measure  of  its  technological  significance,”  U.S.  Patent  Office’s  Office  of 
Technology  Assessment  and  Forecast  Sixth  Report  cited  in  Trajtenberg  (1990, 
pp.  173-174). 

However,  there  are  a  number  of  factors  that  should  obscure  the  relationship  between  forward 
citations  and  knowledge  transfer  or  patent  value.106  First,  forward  citations  occur  for  purely  legal 
reasons  that  have  no  connection  with  actual  use  or  knowledge  transfer  (Eusebi  &  Silberglitt,  n.d.; 
Strumsky  et  al.,  2012).  Second,  a  patent  may  describe  a  concept  that  is  fundamental  to  many 
other  future  technologies,  but  that  specific  patent  may  never  have  been  implemented  in  a 
physical  invention  offered  to  the  market.  Third,  patents  that  are  written  very  broadly  will 
intersect  with  more  patents  than  those  that  are  written  modestly.  Therefore,  the  number  of 
forward  citations  is  confounded  by  the  language  in  the  patent,  which  is  independent  of  the 


106  Patent  forward  citations  are  used  in  three  different  ways  in  the  study  of  research  outputs.  One  approach  is  to 
view  the  network  of  citations  as  a  map  that  traces  the  lineage  of  patented  technologies.  A  second  and  closely  related 
approach  is  to  view  the  network  of  citations  as  a  network  of  collaboration  or  intellectual  influence.  The  third 
approach  estimates  the  significance  or  value  of  a  patent  based  on  the  relative  number  of  citations  it  has  received. 
Only  this  last  approach  is  relevant  to  this  study. 
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usefulness  of  the  patent  (unless  it  turns  out  that  more  concise  patents  are  more  valuable  patents) 
(Eusebi,  2012).  Fourth,  the  number  of  forward  citations  are  also  a  function  of  time.  Older  patents 
have  had  the  opportunity  to  accumulate  more  citations  from  a  larger  pool  of  patents  (Marco, 
2007).  Furthermore,  while  some  studies  have  correlated  forward  citation  counts  with  direct 
measures  of  patent  value  (summarized  below),  other  studies  have  found  that  citation  counts 
explain  a  very  small  fraction  of  the  total  variation  in  patent  value  (Gambardella  et  al.,  2008). 

Despite  these  drawbacks,  there  is  some  evidence  that  the  number  of  forward  citations, 
relative  to  other  patents  for  similar  technologies,  correlates  with  the  commercial  value  of  the 
patent.  Analyzing  patents  related  to  Computed  Tomography  (CT)  scan  devices,  Trajtenberg 
(1990)  correlated  the  number  of  forward  citations  a  patent  received  with  an  estimate  of  the 
increment  of  consumer  surplus  gained  by  the  underlying  invention.  Controlling  for  the  age  of  the 
patent,  more  citations  positively  correlated  with  higher  consumer  surplus  gains  for  the  new 
invention.  Trajtenberg  also  found  that  the  largest  consumer  surplus  gains  occurred  early  in  the 
life  of  the  technology  even  though  industry  R&D  spending  peaked  later. 

B.  H.  Hall  et  al.  (2005)  found  that  while  accumulated  R&D  stocks  were  the  largest  predictor 
of  firm  value,  firms  that  held  highly-cited  patents  had  market  valuations  50%  higher  than  they 
would  have  predicted  based  on  R&D  stocks  or  raw  patent  counts  alone.  Hall  et  al.  say  their  work 
"confirms  that  the  market  values...  'high-quality'  R&D  output  as  measured  by  citation  intensity," 
(B.  H.  Hall  et  al.,  2005,  p.  17).  However,  it  seems  unlikely  that  market  analysts  are  literally 
examining  a  firm's  patent  citation  intensity  and  modifying  stock  recommendations  based  on  that 
analysis.  It  seems  more  likely  that  citation  intensity  correlates  with  some  other  firm  characteristic 
into  which  market  analysts  have  more  visibility.  Supporting  this  objection,  Gu  (2005)  confirms 
that  patent  citations  are  useful  predictors  of  a  firm's  future  earnings,  but  finds  that  investors  do 
not  behave  as  if  they  include  patent  citations  in  their  decision-making.108 

Czamitzki  et  al.  (2011)  further  investigate  the  quality  of  patent  citation  metrics  by  seeing  if 
they  will  distinguish  "wacky"  patents  from  other  patents.  "Wacky"  patents  are  those  identified  by 
www.patentoftheweek.com  as  particularly  futile,  silly,  obvious,  or  incremental.  Indeed,  wacky 
patents  had  significantly  fewer  forward  citations.  However,  wacky  patents  had  references  to  and 
from  patents  in  a  significantly  broader  array  of  technology  classes.  The  authors  say  this  diversity 
in  technological  antecedents  occurs  because  wacky  patents  represent  combinations  of 
technologies  "that  should  not  be  joined  because  the  combination  is  trivial  or  useless,"  (Czamitzki 
et  al.,  201  l,p.  134). 


107  B.  H.  Hall  et  al.  (2005)  use  Tobin's  Q  as  the  measure  of  firm  value.  Tobin's  Q  is,  basically,  the  ratio  of  a  firm's 
market  valuation  to  the  replacement  cost  of  the  firm's  physical  assets.  See  section  on  Patent  valuation  via  stock 
price. 

108 

A  key  exception  to  this  objection  may  be  the  pharmaceutical  industry  in  which  market  analysts  may  very  well 
pay  close  attention  to  patent  characteristics,  although  such  a  model  would  also  have  to  account  for  the  drop  in 
earnings  after  a  drug  patent  expires  and  generics  become  available  (see  for  example  Hodgson,  Falconi,  &  Spencer 
(2012)).  Incidentally,  B.  H.  Hall  et  al.  (2005)  do  find  that  patent  citations  have  the  largest  effect  on  firm  value  in  the 
pharmaceutical  industry. 
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Forward  citations  have  also  been  used  to  evaluate  the  quality  of  federal  laboratory  patents. 
Jaffe  &  Lemer  (2001)  used  forward  citations  to  gauge  changes  in  the  quality  of  patents  fded  by 
DOE  laboratories  following  legislation  in  the  1980s  aimed  at  increasing  transfer  of  technology 
from  federal  laboratories  to  the  private  sector.  They  find  that  patents  awarded  to  DOE  Federally 
Funded  Research  and  Development  Centers  (FFRDCs)  increased  in  the  early  1990s  despite 
falling  research  funding.  While  a  similar  increase  in  patenting  occurred  in  universities  following 
the  Bayh-Dole  act,  universities  appeared  to  be  submitting  lower  quality  patents  as  measured  by 
the  average  number  of  forward  citations  received  (Henderson,  Jaffe,  &  Trajtenberg,  1998).  In 
comparison,  the  quality  of  DOE  patents,  as  measured  by  forward  citations,  remained  nearly  the 
same  as  the  volume  of  patents  increased  (Jaffe  &  Lemer,  2001). 

Jaffe,  Fogarty,  &  Banks  (1997)  used  similar  methods  to  examine  the  quality  of  NASA 
patents  from  the  1970s  through  the  1980s.  This  time  Jaffe  et  al  found  that  NASA  patent  quality, 
as  measured  by  forward  citations,  peaked  in  the  late  1970s  and  declined  over  the  following 
decade,  although  NASA  patents  remained  more  highly  cited  than  the  average  federal  patent.  The 
authors  also  point  out  that  citation-weighted  patent  counts  should  not  be  used  as  the  only 
measure  of  a  NASA  lab's  performance  because  NASA  likely  does  not  seek  patents  for  much  of 
its  work  and  companies  may  not  feel  they  need  to  cite  NASA  patents  because  NASA,  like  most 
federal  agencies,  is  unlikely  to  sue  for  infringement.  Both  behaviors  would  cause  citation- 
weighted  patent  counts  to  under  estimate  the  innovativeness  of  a  NASA  facility. 

Drawbacks  to  indirect  methods 

The  main  drawback  to  using  renewal  data  to  judge  the  innovativeness  of  DOD  patents  is  the 
truncation  problem  described  above.  A  second  drawback  is  that  the  renewal  rate  may  be 
determined  more  by  the  quality  of  the  ORTA  than  the  quality  of  the  research  since  the  decision 
to  renew  will  be  based  on  the  ORTA’s  understanding  of  commercialization  opportunities.  A  third 
drawback  is  that  the  DOD  may  have  an  incentive  to  let  patents  lapse  quickly  so  the  information 
is  in  the  public  domain.  After  all,  commercial  profit  is  not  the  primary  motive  for  patenting  in  the 
DOD,  and  placing  patented  inventions  in  the  public  domain  can  support  many  of  their  other 
objectives  including  preventing  contractors  from  claiming  the  invention  and  charging  more  for  it 
and  encouraging  the  use  of  new  technologies  in  the  defense  industry  (see  the  pipe  coatings  story 
in  the  “Drawbacks  to  survey  methods”  section).  For  all  of  these  reasons,  patent  renewal  rates  are 
not  a  good  measure  of  the  quality,  much  less  the  innovativeness  of  government  patents. 

If  forward  citations  correlate  with  commercial  patent  value,  then  forward  citations  could  be  a 
good  approach  to  finding  innovative  government  patents.  Indeed,  Jaffe  et  al.  (1997)  and  Jaffe  & 
Lemer  (2001)  explore  this  approach  for  NASA  and  DOE  patents.  Some  controls  would  be 
needed  for  industry,  technology  area,  and  age  of  the  patent,  but  these  adjustments  are  feasible.  In 
addition,  relative  citation  rates  may  only  be  comparable  with  other  government  patents,  rather 
than  across  the  entire  industry,  because  of  the  defensive  issues  described  in  Jaffe  et  al.  (1997). 
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However,  as  mentioned  in  the  second  paragraph  of  this  section,  there  are  several  reasons  to 
be  wary  of  forward  citations  as  a  measure  of  patent  value.  Gambardella  et  al.  (2008)  found  that 
the  number  of  forward  citations  accounted  for  less  than  3%  of  the  variation  in  patent  values. 
Perhaps  more  importantly,  the  theoretical  basis  for  using  forward  citations  as  a  measure  of  patent 
value  -  that  highly  cited  patents  represent  more  fundamental  technologies  that  are  built  upon  by 
later  patents  -  is  in  some  conflict  with  the  actual  practice  of  patent  law. 

Patent  citations  occur  for  legal  reasons  that  are  unrelated  to  attribution  including  the 
responsibility  of  the  patent  examiner  to  skeptically  test  the  novelty  of  the  patent’s  claims 
(Alcacer,  Gittelman,  &  Sampat,  2009;  Eusebi  &  Silberglitt,  n.d.;  Jaffe  et  al.,  1997;  Strumsky  et 
al.,  2012).  Studies  that  have  attempted  to  assert  a  more  literal  connection  between  the  citing  and 
cited  work  have  found  a  great  deal  of  noise  in  the  citation  signal.  Based  on  interviews  with 
inventors,  fewer  than  50%  of  patent  citations  seem  to  reflect  to  an  extension  of  the  cited 
technology  area  or  correspond  with  any  known  communication  between  inventors  (Jaffe  et  al., 
1997;  Jaffe,  Trajtenberg,  &  Fogarty,  2000;  Packalen  &  Bhattacharya,  2012).  This  finding  is 
unsurprising  when  one  considers  that  the  majority  of  citations  on  the  average  patent  are  added  by 
the  patent  examiner  (Alcacer  et  al.,  2009). 

B.3  Technology  classification-based  approaches  to  patent  valuation 

Recognizing  the  limitations  of  citation-based  estimates  of  patent  value,  some  authors  are 
beginning  to  explore  whether  relationships  within  the  network  of  USPTO  patent  technology 
classes  are  better  descriptions  of  idea  networks  and,  furthermore,  whether  some  of  these 
relationships  can  help  identify  innovations.  Packalen  &  Bhattacharya  (2012)  use  text  analysis  of 
patent  claims  and  the  USPTO  technology  classifications  to  identify  early  work  in  a  technology 
area  and  to  separate  citations  that  indicate  similar  claims  from  citations  that  do  indicate  the 
building  of  ideas.  Strumsky  et  al.  (2012)  recommend  using  the  patent  technology  classification 
system  to  explore  the  evolution  of  the  components  of  technologies.  Strumsky  et  al.  argue  that 
patents  actually  represent  bundles  of  “technological  capabilities,”  each  of  which  has  its  own 
history,  and  that  citation-based  studies  therefore  provide  only  a  coarse  look  at  the  evolution  of 
technologies.  Eusebi  &  Silberglitt  (N.d.)  actually  do  trace  the  evolution  of  technologies  through 
the  co-occurrence  of  technology  classes  within  patents.  Eusebi  and  Silberglitt  furthermore  use 
the  existence  of  s-curves  in  the  accumulation  of  patents  within  technology  subclasses  to  identify 
emerging  technologies  and  their  antecedents. 

This  study  builds  on  Eusebi’ s  technology  class  s-curve  work  to  construct  indicators  of  patent 
value  that  are  based  entirely  within  the  technology  classification  signal.  Additional  discussion  of 
the  relationships  between  s-curves  in  the  patent  technology  class  system  and  patent  value  or 
innovativeness  can  be  found  in  section  2.3. 
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Appendix  C  USPTO  Classification  System 


Since  1900,  the  U.S.  Patent  and  Trademark  Office  (USPTO)  has  maintained  a  classification 
system  to  organize  and  describe  the  subject  matter  of  patent  documents  (USPTO,  2012b).  This 
system  evolved  to  help  patent  examiners  group,  search  for,  and  more  quickly  identify  other 
patents  as  they  consider  the  validity  of  a  new  patent  application  (Eusebi,  2012),  but  this  system 
can  also  be  used  by  researchers  to  monitor  the  evolution  of  technologies. 

Patent  examiners  assign  each  patent  they  process  to  one  primary  (aka  original)  and  any 
number  of  additional  (called  cross  reference)  technology  classes  (Strumsky  et  al.,  2012;  USPTO, 
2012c).  This  classification  summarizes  all  of  the  distinct  technologies  and  applications  described 
in  a  patent’s  claims  (Strumsky  et  al.,  2012). 109  A  technology  class  must  be  identified  for  each 
claim,  although  if  more  than  one  claim  falls  under  the  same  class  the  assignment  is  only  listed 
once  (Strumsky  et  al.,  2012;  USPTO,  2012d).  The  rules  for  placing  a  patent  claim  in  the 
appropriate  class  are  described  in  detail  in  USPTO  (201 2d).  Strumsky  et  al.  (2012)  provide 
several  examples  illustrating  how  a  patent’s  many  claims  can  be  reduced  to  a  few  technology 
class  assignments.  Strumsky  et  al.  note  that  there  are  many  fewer  class  assignments  on  average, 
4.4  per  patent  in  2005,  than  there  are  claims  per  patent,  which  they  uses  as  evidence  for  the 
“descriptive  efficiency  of  the  technology  codes,”  (Strumsky  et  al.,  2012,  p.  8). 

Each  technology  class  assignment  is  listed  on  the  front  page  of  a  patent  as  a  pair  of  numbers, 
e.g.  501/137.  The  number  before  the  slash  indicates  the  top-level  technology  class  and  the 
number  after  the  slash  identifies  the  technology  subclass.  The  top-level  technology  classes,  of 
which  there  are  approximately  500,  are  broad  descriptions  of  technologies  like  “Power  Plants”, 
“Ordnance”,  “Television”,  or  “Organic  Compounds”).  These  classes  are  also  arranged  into  about 
30  subject  groups  (USPTO,  2012e)  which  provide  a  good  initial  overview  of  the  classification 
system  (USPTO,  201 2d). 

Subclasses,  of  which  there  are  around  150, 000, 110  form  a  hierarchical  system  that  gives  more 
detail  about  the  “processes,  structural  features,  and  functional  features”  (USPTO,  2012c,  p.  1)  of 
the  technology  described  in  the  patent.  For  example,  to  find  the  description  of  class  501 
(Compositions:  Ceramic)  subclass  137  the  reader  must  trace  a  6-level  chain: 

>  Level  I:  Class  501:  Compositions:  Ceramic 

>  Level  II:  Subclass  1:  Ceramic  Compositions: 


109 

“[A  patent’s]  claims  state,  in  technical  and  precise  terms,  the  subject  matter  of  the  invention  (or  discovery),  as 
well  as  its  purpose,  principal  properties,  how  it  operates  and  the  methods  it  employs,”  (Strumsky  et  al.,  2012,  p.  4). 

110  Based  on  an  analysis  of  classes  and  subclasses  listed  in  the  Jan.  6,  2012  USPTO  master  classification  file. 
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>  Level  III:  Subclass  134:  Titanate,  zirconate,  stannate,  niobate,  or  tantalate 
or  oxide  of  titanium,  zirconium,  tin,  niobium,  or  tantalum  containing  (e.g., 
dielectrics,  etc.): 

>  Level  IV:  Subclass  135:  Alkaline  earth  or  magnesium  containing: 

>  Level  V:  Subclass  136:  Titanate  containing: 

>  Level  VI:  Subclass  137:  Barium  titanate: 

Each  entry  on  this  chain  is  included  in  the  list  of  subclasses.  Because  they  are  meant  to  be  read 
hierarchically,  any  given  subclass  title  may  be  difficult  to  comprehend  out  of  context.  The  full 
class-to-subclass  path,  however,  is  too  long  to  use  frequently  in  a  discussion  or  as  a  label  on  a 
chart.  Therefore,  this  study  has  created  more  lay-person- friendly  short  description  of  classes  and 
subclasses.  Using  the  above  example,  this  text  refers  to  class  501/137  as  Ceramics  Containing 
Barium  Titanate. 

The  patent  office  develops  new  classes  as  new  areas  of  technology  emerge.  For  example, 
class  977,  nanotechnology,  was  created  in  2004.  When  new  technology  classes  are  defined, 
patent  examiners  reclassify  older  patents  into  the  new  category  and  retire  some  older  obsolete 
classes.  Therefore,  the  first  nanotechnology  patent  appears  in  1986  (Strumsky  et  al.,  2012).  The 
USPTO  issues  monthly  “Classification  Orders”,  which  are  available  at  the  USPTO  website, 
detailing  changes  to  the  technology  class  system. 
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Appendix  D  Overview  of  the  DOD  laboratory  system 


This  section  provides  an  overview  of  the  Army,  Navy,  and  Air  Force  science  and  technology 
(S&T)  systems.  It  provides  a  summary  of  each  laboratory’s  budget  and  technological  focus  area. 
The  relationship  between  funding  and  technology  areas  with  patenting  patterns  is  explored  in 
chapter  5. 

As  stated  in  chapter  3,  this  study  only  covers  the  in-house  S&T  research  laboratories.  S&T  is 
defined  by  budget  activities  6.1  (basic  research),  6.2  (applied  research),  and  6.3  (technology 
development  research).111  Contractor-operated  research  facilities  and  doctrine-focused 
“laboratories”  like  the  Battlelabs  are  excluded.  With  this  restriction  in  mind,  each  Service’s 
laboratory  system  can  be  described  as  a  combination  of  four  types  of  organizations: 

1 .  a  “corporate”  laboratory  that  performs  basic  and  applied  research  in  a  wide  variety  of 
fields, 

2.  several  applied  research  and  technology  development  product  centers  focused  on  a  few 
fields  of  technology  each, 

3.  a  medical  research  command  with  subordinate  laboratories  and  hospitals,  and 

4.  an  extramural  basic  research  office  that  primarily  funds  university  researchers 

In  practice,  each  service  apportions  its  research  funds  differently,  with  different 
organizational  layers  between  the  laboratory  researchers  and  the  Service  Secretary,  and  with 
some  research  performing  organizations  also  writing  external  grants  to  various  government  and 
private- sector  research  groups.  Each  of  the  following  three  sub-sections  illustrates  how  each 
service  organizes  these  four  functions  into  different  organizational  units,  lists  the  technological 
focus  areas  of  each  laboratory,  and  summarizes  each  laboratory’s  relative  emphasis  on  basic, 
applied,  or  technology  development  research.  Later  sections  of  this  chapter  will  show  how 
technology  focus  area  and  the  distribution  of  basic,  applied,  and  technology  development 
research  correlate  with  a  lab’s  patent  trends. 

Note  that  an  organization  is  included  in  this  study  its  budget  and  personnel  numbers  are 
reported  separately  in  the  2007  DOD  In-House  S&T  Activities  Report.  However,  some 
“laboratories”  are  aggregated  or  disaggregated  based  on  the  availability  of  patent  or  budget  data 
for  that  organizational  unit  (see  chapter  4).  As  a  result,  some  organizations  that  should  more 

1 1 1  The  DOD  allocates  research  and  development  money  according  to  a  sequence  of  categories  that  loosely 
represent  a  linear  view  of  technology  development.  The  entire  category  is  called  Research,  Development,  Test  and 
Evaluation  (RDT&E).  The  first  three  levels  are  collectively  referred  to  as  science  and  technology  (S&T).  S&T 
includes  basic  research,  applied  research,  and  technology  development.  DOD  budget  documents  refer  to  these 
activities  by  the  numbers  6.1  (basic),  6.2  (applied),  and  6.3  (technology  development),  and  discussions  of  DOD 
RDT&E  often  use  the  numbers  rather  than  the  words.  The  remainder  of  the  RDT&E  category  consists  of  Advanced 
Component  Development  and  Prototypes  (ACD&P)  (6.4),  System  Development  and  Demonstration  (SDD)  (6.5), 
RDT&E  Management  Support  (6.6),  and  Operational  System  Development  (6.7).  See  DOD  regulation  7000. 14-R, 
Volume  2B,  Chapter  5  (Office  of  the  Comptroller,  2008)  for  the  formal  definitions  of  each  category. 
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properly  be  thought  of  as  commands  (i.e.  large  multi-function  offices  that  include  things  like 
acquisition  and  logistics  as  well  as  the  operation  of  one  or  more  laboratories),  particularly  the 
medical  research  commands,  are  described  here  as  one  laboratory. 

D.1  Air  Force  laboratory  system 

Among  the  Army,  the  Navy,  and  the  Air  Force,  the  Air  Force  has  the  most  straightforward 
S&T  funding  system.  Since  1997,  all  Air  Force  S&T  funding  has  been  directed  through  the  Air 
Force  Research  Lab  (AFRL)  (Defense  Science  Board,  2012).  AFRL  contains  nine  research¬ 
performing  directorates  organized  by  technology  area  plus  the  grant-issuing  Air  Force  Office  of 
Scientific  Research  (AFOSR).  This  basic  organizational  structure  is  shown  in  Figure  D.l.  The 
directorates’  focus  areas  are  listed  in  table  D.l. 

AFRL  is  a  relatively  recent  organizational  construct  that  administratively  groups  many 
formerly  independent  Air  Force  research  organizations.  AFRL  is  headquartered  at  Wright- 
Patterson  Air  Force  Base  in  Ohio,  but  some  of  the  directorates  are  physically  located  in  New 
Mexico,  New  York,  Florida,  or  Virginia  reflecting  the  physical  dispersion  of  Air  Force  research 
prior  to  the  consolidation  of  AFRL  in  the  mid-1990s  (see  the  AFRL  organization  chart  available 
at  the  AFRL  website  and  Duffner  (2000)). 

Figure  D.2  shows  the  S&T  funds  spent  in-house  and  directed  out-of-house  for  each  of  the 
AFRL  directorates  as  of  fiscal  year  (FY)  2007,  the  most  recent  year  available  (all  budget  number 
from  U.S.  Department  of  the  Air  Force  (2007)).  Although  Figure  D.2  shows  a  small  amount  of 
in-house  basic  research  at  AFOSR,  AFOSR  is  generally  not  considered  to  be  a  research 
performer.  AFOSR  manages  all  of  the  Air  Force’s  basic  research  funding.  Since  most  of  this 
funding  is  issued  for  university  research  projects,  AFOSR  could  be  considered  the  Air  Force’s 
extramural  research  funding  agency,  but  AFOSR  also  disperses  a  small  amount  of  basic  research 
funds  to  other  AFRL  directorates  (Defense  Science  Board,  2012). 

AFRL’s  research  funding  is  primarily  designated  for  applied  research  or  technology 
development  and  primarily  goes  to  external  research  contracts.  External  research  contracts  make 
up  at  least  50%  of  each  of  the  directorate’s  S&T  portfolio.  In  AFRL  RD  (directed  energy)  and 
AFRL  RV  (space  vehicles),  external  research  contracts  make  up  over  three-quarters  of  the  total 
S&T  funding  managed  by  the  directorate.  Within  their  in-house  portfolios,  the  AFRL 
directorates  are  primarily  applied  research  laboratories.  Between  50  and  90  percent  of  each 
directorate’s  in-house  portfolio  is  applied  research,  technology  development  research  accounts 
for  7  to  50  percent  of  in-house  research,  while  basic  research  makes  up  between  1  and  10 
percent.112 


1 12 

One  commenter  notes  that  AFRL  pays  for  the  lab’s  management  activities  out  of  its  applied  research  account  and 
that  this  is  not  necessarily  the  practice  of  the  other  service  research  labs. 
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Figure  D.1  Major  RDT&E  organizations  -  Air  Force 
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Lab  (in  order  of  In-House  funding) 


Figure  D.2  Air  Force  laboratory  S&T  funding  FY2007 
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Table  D.1  AFRL  directorate  descriptions 


Acronym 

Full  Name 

Focus  Areas 

AFOSR 

Air  Force  Office  of  Scientific  Research 

Various,  extramural 

AFRL  711th 

711th  Human  Performance  Wing 

Human  factors,  aerospace  medicine 

AFRL  RB 

Air  vehicles 

Aeronautics,  control  systems,  structures, 
systems  integration 

AFRL  RD 

Directed  Energy 

Laser  systems,  high  power  electromagnetics, 
weapons  modeling  and  simulation,  directed 
energy,  electro-optics 

AFRL  Rl 

Information 

Communications  and  networking,  modeling 
and  simulation,  information  warfare, 
information  fusion 

AFRL  RV 

Space  vehicles 

Radiation-hardened  electronics;  space 
power,  structures  and  control;  in-space 
propulsion;  space  environment 

AFRL  RW 

Munitions 

Air-launched  munitions 

AFRL  RX 

Materials  and  Manufacturing 

Material  science,  manufacturing  processes 

AFRL  RY 

Sensors 

Radio  frequency  and  electro-optic  sensing, 
sensor  fusion,  network-enabled  spectrum 
warfare 

AFRL  RZ 

Propulsion 

Aircraft  and  rocket  engines,  power  sources 

Source:  Laboratory  websites 

D.2  Army  laboratory  system 

The  Army  may  have  the  most  disaggregated  laboratory  system  of  the  services.  The  Army 
splits  S&T  funding  and  program  management  between  the  Army  Research  Lab  (ARL),  five 
Research,  Development,  and  Engineering  Centers  (RDECs),  the  Army  Corps  of  Engineers' 
laboratory  (ERDC),  the  U.S.  Army  Medical  Research  and  Materiel  Command  (USAMRMC), 
and  a  few  other  miscellaneous  centers  (Defense  Science  Board,  2012).  Figure  D.3  shows  the 
distribution  of  these  research  centers  under  various  Army  commands.  Note  that  USAMRMC  and 
ERDC  have  subordinate  organizations  that  are  sometimes  listed  as  individual  laboratories. 
TechMatch  did  not  identify  patents  for  those  component  laboratories  individually.  Also,  while 
they  are  sometimes  included  in  discussions  of  the  Army’s  S&T  system  (e.g.  in  Miller  (2010))  the 
Army  Research  Institute  (AR1)  for  the  Behavioral  and  Social  Sciences,  the  Space  and  Missile 
Defense  Technical  Center  (SMDTC),  and  the  Simulation  and  Training  Technology  Center 
(STTC),  have  less  than  $12  million  in  S&T  funding  as  of  2007.  Because  TechMatch  includes  no 
patent  data  for  these  labs,  they  are  excluded  in  the  following  discussions. 

Figure  D.4  shows  the  S&T  funds,  as  of  fiscal  year  (FY)  2007,  spent  in-house  and  directed 
out-of-house  for  each  Army  laboratory  included  in  this  study  (all  budget  data  from  (U.S. 
Department  of  the  Army,  2007)).  Around  75%  of  the  Army’s  S&T  funding  flows  through  ARL 
and  the  five  RDECs,  ECBC,  ERDC,  and  USAMRMC  each  account  for  about  6%. 
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ARL  is  the  Army’s  “corporate  laboratory”  and  the  Army’s  primary  performer  of  general 
basic  research.  USAMRMC  has  a  larger  amount  of  in-house  basic  research  ($67M  in  2007 
versus  ARL’s  $44M),  but  USAMRMC’s  mission  is  focused  on  medical  R&D.  Approximately 
one-fifth  of  ARL’s  in-house  research  budget  is  basic  while  the  rest  is  applied.  Note  that  while  the 
major  extramural  basic  research  office  is  broken  out  separately  in  the  Air  Force  and  Navy 
budgets,  the  Army  Research  Office  (ARO)  sits  within  ARL  and  is  not  reported  separately. 
Therefore,  ARL’s  extramural  budget  is  not  directly  comparable  with  the  extramural  research 
budget  at  NRL  or  the  non-AFOSR  AFRL  directorates. 

The  S&T  portfolios  of  the  five  RDECs  (AMRDEC,  ARDEC,  CERDEC,  TARDEC,  and 
NSRDEC)  focus  on  applied  research  and  technology  development,  although  they  also  have 
substantial  budgets  for  technology  demonstration  through  operational  system  development  (i.e. 
6.4  through  6.7,  not  shown  in  Figure  D.4).  The  RDECs  can  be  thought  of  as  the  direct  customers 
of  ARL’s  basic  and  applied  research  (Lyons,  Mait,  &  Schmidt,  2005).  While  ARL  is  supposed  to 
develop  fundamental  technologies,  the  RDECs  are  responsible  for  proving  and  developing  those 
technologies  for  military  applications  (Wong,  2003).  While  the  RDECs  have  small  amount  of 
basic  research  funding  through  the  Independent  Laboratory  Innovation  Research  (ILIR)  program 
(Decker  et  al.,  2010;  Lyons  et  al.,  2005),  the  near  monopoly  of  basic  research  funding  at  ARL 
reflects  the  intended  division  of  labor. 

In  ARL  and  the  Army  RDECs,  around  70  to  80  percent  of  S&T  work  is  contracted  out-of- 
house.  USAMRMC  executes  very  few  extramural  research  contracts,  and  the  Army  Corps  of 
Engineer’s  ERDC  outsources  slightly  less  than  50%  of  its  S&T  work.  Overall,  both  the  Army 
and  the  Air  Force  (including  AFOSR)  split  their  S&T  funding  30%  in-house,  70%  out-of-house. 
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Figure  D.3  Major  RDT&E  organizations  -  Army 


Sources:  Carroll  FedSearch  (accessed  12/6/2012),  U.S.  Department  of  the  Army  (2007),  Miller  (2010) 
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Lab  (in  order  of  In-House  funding) 


Figure  D.4  Army  laboratory  S&T  funding  FY2007 
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Table  D.2  Army  laboratory  descriptions 


Acronym  Full  Name 


Focus  Areas 


AMRDEC 

ARDEC 

ARL 

CERDEC 

ECBC 


Aviation  and  Missile  Research,  Development 
and  Engineering  Center 

Armament  Research,  Development  and 
Engineering  Center 

Army  Research  Laboratory 

Communications-Electronics  Research, 
Development  and  Engineering  Center 

Edgewood  Chemical  Biological  Center 


Air  vehicles,  missiles 

Armaments 

Various 

C4ISR 

Non-medical  chemical  and  biological  (CB) 
defense 


ERDC 


Engineer  Research  and  Development  Center  Army  Corps  of  Engineers,  structures, 

information  technology,  mapping,  operating 
in  various  environments 


NSRDEC 


Natick  Soldier  Research,  Development  and 
Engineering  Center 


TARDEC 


USAMRMC 


Tank  Automotive  Research,  Development 
and  Engineering  Center 

U.S.  Army  Medical  Research  and  Materiel 
Command 


Source:  Laboratory  websites 


Food,  clothing,  human  factors,  biotech, 
materials,  lasers,  magnetic  resonance 
imaging 

Armor,  tanks 
Medical 


D.3  Navy  laboratory  system 

Unlike  in  the  Army  and  the  Air  Force,  in  the  Navy  the  organization  that  manages  extramural 
basic  research  grants  sits  outside  the  performing  laboratories.  The  Navy’s  S&T  system  consists 
primarily  of  the  Office  of  Naval  Research  (ONR),  the  Naval  Research  Laboratory  (NRL),  and 
laboratories  within  the  four  Navy  Warfare  Centers.  ONR  manages  the  Navy's  extramural  basic 
and  applied  research  programs,  including  a  basic  research  pass-through  to  NRL.  NRL  is  the 
Navy's  "corporate  laboratory"  and  primarily  conducts  in-house  basic,  applied,  and  development 
research  for  Navy  customers.  NRL  also  contracts  work  out  to  other  research  entities  and  private 
companies  in  the  course  of  achieving  its  mission.  While  NRL  reports  to  the  Office  of  Naval 
Research  (ONR),  the  Navy  Warfare  Centers  report  to  the  System  Commands  (the  Naval  Sea 
Systems  Command  (NAVSEA),  the  Naval  Air  Systems  Command  (NAVAIR),  and  the  Space 
and  Naval  Warfare  Systems  Command  (SPAWAR)),  which  manage  the  Navy’s  acquisition 
programs.  Consequently,  the  Warfare  Centers  tend  to  be  more  focused  on  the  immediate  needs  of 
the  fleet  (see  discussion  of  the  evolution  of  the  Warfare  Centers  in  Hazell  (2008)).  Figure  D.5 
shows  the  position  of  each  organization  within  the  Navy’s  S&T  system,  and  table  D.3  describes 
the  technological  focus  of  each  laboratory.  Figure  D.6  shows  the  distribution  of  S&T  funding  for 
NRL,  the  Warfare  Centers,  and  the  Navy  Medical  Research  Command. 

NRL,  as  the  Navy’s  corporate  research  lab,  has  a  portfolio  that  is  nearly  evenly  balanced 
among  basic,  applied,  and  development  research.  Unlike  the  AFRL  directorates  and  ARL,  NRL 
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actually  contracts  out  less  work  than  it  conducts  in-house,  although  recall  that  ARL’s  extramural 
budget  includes  ARO  which  skews  the  comparison.  The  degree  of  out-of-house  contracting  at 
the  Warfare  Centers  varies,  but  overall  the  Navy  S&T  budget  is  split  about  50/50  in-house/out- 
of-house. 

Like  the  Army  RDECs,  the  Naval  Warfare  Centers  have  S&T  budgets  that  heavily  lean 
towards  applied  research  and  development  in  addition  to  large  amounts  of  post-technology 
development  (i.e.  6.4  through  6.7)  funding  (see  Hazell  (2008),  the  In-House  S&T  Activities 
report  does  not  give  post-S&T  funding  for  the  Warfare  Centers).  Research  and  development  is 
actually  in  the  minority  of  the  Warfare  Centers’  overall  activities,  accounting  for  somewhere 
between  5  and  30  percent  of  their  total  business  base  (reports  vary,  see  Hazell  (2008)  and 
Saunders  et  al.  (1995)).  While  patents  were  reported  for  specific  research  laboratories  operated 
by  the  Warfare  Centers  (see  laboratories  listed  to  the  far  right  in  Figure  D.5),  separate  budgets 
were  not.  Note  that  patents  are  not  available  for  all  of  the  Warfare  Center  laboratories.  Those 
Warfare  Centers  lacking  patents  are  also  those  that  arguably  should  not  be  defined  as  “in-house 
laboratories”  because  they  focus  primarily  on  training,  testing,  or  servicing  systems  installed  on 
ships  rather  than  on  technology  development  per  se  (see  table  D.3). 

All  of  the  Navy’s  in-house  research  and  development  work  is  primarily  funded  via  contracts 
with  customers.  This  system  is  called  “Working  Capital  funding”  because  the  exchanges  of  funds 
in  these  agreements  are  handled  through  the  Navy’s  Working  Capital  account  (see  Department  of 
the  Navy  (2012))This  system  evolved  in  the  Navy  because  the  various  labs  traditionally  did  work 
for  many  different  commands  and  even  other  Services  rather  than  being  beholden  to  one  superior 
headquarters  organization.  Paying  for  research  services  out  of  the  Working  Capital  fund  fairly 
distributed  the  total  costs  of  supporting  the  laboratories  to  all  customers  (Hazell,  2008). 
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Figure  D.5  Major  RDT&E  organizations  -  Navy 


Sources:  Carrol  FedSearch  (accessed  12/6/2012),  U.S.  Department  of  the  Navy  (2007),  Warfare  Center  websites 

(accessed  12/6/2012) 
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Lab  (in  order  of  In-House  funding) 


Figure  D.6  Navy  laboratory  S&T  funding  FY2007 
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Table  D.3  Navy  laboratory  descriptions 


Laboratory 

Parent  Organization 

Patents 

Focus  Areas 

NAWC  Patuxent  River 

Naval  Air  Warfare  Center 

Yes 

Life-cycle  support  of  manned  and 
unmanned  aircraft  and  components 

NAWC  China  Lake 

Naval  Air  Warfare  Center 

Yes 

Weapons  Division's  land  range,  missiles 
and  bombs,  avionics,  basic  and  applied 
research,  integration  and  testing 

NAWC  Point  Mugu 

Naval  Air  Warfare  Center 

Yes 

Weapons  Division's  sea  range,  special 
weapons  and  projects,  technology 
transfer,  electronic  warfare,  integration 
and  testing 

Naval  Medical  Research 
Center  (NMRC) 

Yes 

Medical 

NSWC  Carderock 

Naval  Surface  Warfare  Center 

Yes 

Ships  and  ship  systems 

NSWC  Corona 

Naval  Surface  Warfare  Center 

No 

System  test  and  assessment, 
measurement  science  and  calibration 
standards 

NSWC  Crane 

Naval  Surface  Warfare  Center 

No 

Life-cycle  support  for  sensors,  electronics, 
electronic  warfare,  and  special  operations 

NSWC  Dahlgren 

Naval  Surface  Warfare  Center 

Yes 

Weapon  system  integration 

NSWC  Dam  Neck 

Naval  Surface  Warfare  Center 

No 

Training,  Integrated  Combat  Control 
Systems,  Information  Operations 

NSWC  Indian  Head 

Naval  Surface  Warfare  Center 

Yes 

Explosives,  propellants,  pyrotechnics  and 
their  immediately  related  components 

NSWC  Panama  City 

Naval  Surface  Warfare  Center 

No 

Mine  warfare  systems,  mines,  naval 
special  warfare  systems,  diving  and  life 
support  systems,  amphibious  and 
expeditionary  maneuver  warfare  systems, 
other  missions  that  occur  primarily  in 
coastal  (littoral)  regions 

NSWC  Pt.  Hueneme 

Naval  Surface  Warfare  Center 

No 

Surface  ship  combat  system  engineering 

NUWC  Newport 

Naval  Undersea  Warfare 

Center 

Yes 

Submarines,  autonomous  underwater 
systems,  and  offensive  and  defensive 
undersea  weapon  systems 

NUWC  Keyport 

Naval  Undersea  Warfare 

Center 

No 

Fleet  readiness  support 

NRL 

Office  of  Naval  Research 

Yes 

Various 

SPAWAR  Systems 

Center,  Pacific 

Space  and  Naval  Warfare 
Systems  Command  (SPAWAR) 

Yes 

C4ISR 

SPAWAR  Systems  Space  and  Naval  Warfare 

Center,  Atlantic  Systems  Command  (SPAWAR) 

Source:  Laboratory  websites 

No 

C4ISR 
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